{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "08f4321d-d32a-4a90-bfc7-e923f316b2f8",
   "metadata": {},
   "source": [
    "<table style=\"width:100%\">\n",
    "<tr>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<font size=\"2\">\n",
    "Supplementary code for the <a href=\"http://mng.bz/orYv\">Build a Large Language Model From Scratch</a> book by <a href=\"https://sebastianraschka.com\">Sebastian Raschka</a><br>\n",
    "<br>Code repository: <a href=\"https://github.com/rasbt/LLMs-from-scratch\">https://github.com/rasbt/LLMs-from-scratch</a>\n",
    "<br>汉化的库: <a href=\"https://github.com/GoatCsu/CN-LLMs-from-scratch.git\">https://github.com/GoatCsu/CN-LLMs-from-scratch.git</a>\n",
    "</font>\n",
    "</td>\n",
    "<td style=\"vertical-align:middle; text-align:left;\">\n",
    "<a href=\"http://mng.bz/orYv\"><img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/cover-small.webp\" width=\"100px\"></a>\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ce9295b2-182b-490b-8325-83a67c4a001d",
   "metadata": {},
   "source": [
    "# 第四章: 从零开始构建 GPT 模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f9eac223-a125-40f7-bacc-bd0d890450c7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "matplotlib version: 3.9.0\n",
      "torch version: 2.4.0\n",
      "tiktoken version: 0.7.0\n"
     ]
    }
   ],
   "source": [
    "from importlib.metadata import version\n",
    "\n",
    "import matplotlib\n",
    "import tiktoken\n",
    "import torch\n",
    "\n",
    "print(\"matplotlib version:\", version(\"matplotlib\"))\n",
    "print(\"torch version:\", version(\"torch\"))\n",
    "print(\"tiktoken version:\", version(\"tiktoken\"))\n",
    "#加载并确认版本"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7da97ed-e02f-4d7f-b68e-a0eba3716e02",
   "metadata": {},
   "source": [
    "- 在这一章我们要用类GPT LLM架构\n",
    "- 下一章就是训练LLM了"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d4f11e0-4434-4979-9dee-e1207df0eb01",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/01.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53fe99ab-0bcf-4778-a6b5-6db81fb826ef",
   "metadata": {},
   "source": [
    "## 4.1 LLM架构"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad72d1ff-d82d-4e33-a88e-3c1a8831797b",
   "metadata": {},
   "source": [
    "- 第 1 章讨论了 GPT 和 Llama 等模型，这些模型基于原始 Transformer 架构的解码器部分，按顺序生成单词。\n",
    "- 因此，这些大语言模型（LLM）通常被称为“类解码器”的 LLM。\n",
    "- 与传统的深度学习模型相比，LLM 的规模更大，这主要是由于其参数数量庞大，而非代码量的增加。\n",
    "- 我们将看到，在 LLM 的架构中，许多元素是重复的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c5213e9-bd1c-437e-aee8-f5e8fb717251",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/02.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d43f5e2-fb51-434a-b9be-abeef6b98d99",
   "metadata": {},
   "source": [
    "- 在前几章中，为了便于说明，我们使用了较小的嵌入维度对标记输入和输出进行处理，以确保内容能够显示在单页内。\n",
    "- 本章将讨论与小型 GPT-2 模型类似的嵌入和模型规模。\n",
    "- 我们将具体实现最小的 GPT-2 模型架构（1.24 亿参数）。该架构来源于 Radford 等人的报告 [《Language Models are Unsupervised Multitask Learners》](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)（需要注意的是，初始报告中参数数量被列为 1.17 亿，但这一错误在模型权重库中已被更正）。\n",
    "- 第 6 章将展示如何将预训练权重加载到我们的实现中，这些权重可兼容 3.45 亿、7.62 亿和 15.42 亿参数规模的模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21baa14d-24b8-4820-8191-a2808f7fbabc",
   "metadata": {},
   "source": [
    "- 123million参数的GPT-2配置如下:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5ed66875-1f24-445d-add6-006aae3c5707",
   "metadata": {},
   "outputs": [],
   "source": [
    "GPT_CONFIG_124M = {\n",
    "    \"vocab_size\": 50257,    # Vocabulary size\n",
    "    \"context_length\": 1024, # Context length\n",
    "    \"emb_dim\": 768,         # Embedding dimension\n",
    "    \"n_heads\": 12,          # Number of attention heads\n",
    "    \"n_layers\": 12,         # Number of layers\n",
    "    \"drop_rate\": 0.1,       # Dropout rate\n",
    "    \"qkv_bias\": False       # Query-Key-Value bias\n",
    "}\n",
    "#初始化定义需要的各种超参数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12fcd28-d210-4c57-8be6-06cfcd5d73a4",
   "metadata": {},
   "source": [
    "- 我们使用简短的变量名，以避免代行过长。\n",
    "- `\"vocab_size\"` 表示词汇表大小，共 50,257 个单词，由第 2 章介绍的 BPE 分词器支持。\n",
    "- `\"context_length\"` 表示模型的最大输入标记数，依赖于第 2 章的位置信息嵌入。\n",
    "- `\"emb_dim\"` 是标记输入的嵌入维度，将每个标记转换为 768 维向量。\n",
    "- `\"n_heads\"` 是多头注意力机制中的注意力头数量，详见第 3 章。\n",
    "- `\"n_layers\"` 是模型中 Transformer 块的数量，后续章节会详细实现。\n",
    "- `\"drop_rate\"` 是 dropout 机制的强度，设置为 0.1，表示训练时丢弃 10% 的隐藏单元以防止过拟合（第 3 章讨论）。\n",
    "- `\"qkv_bias\"` 决定多头注意力机制中的 `Linear` 层是否包含偏置向量。现代 LLM 通常禁用此选项，但在第 5 章加载 OpenAI 的 GPT-2 预训练权重时，会重新启用以保持兼容性。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4adce779-857b-4418-9501-12a7f3818d88",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/03.webp\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "619c2eed-f8ea-4ff5-92c3-feda0f29b227",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class DummyGPTModel(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
    "        # 词嵌入层，将输入索引转换为词向量，词表大小由字典大小和特征维度决定。\n",
    "        self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n",
    "        # 位置信息嵌入层，基于文本长度和特征维度生成位置信息。\n",
    "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
    "        # Dropout 层，用于随机丢弃一部分嵌入信息以减少过拟合。\n",
    "\n",
    "        # 使用多个 Transformer 块（占位符）\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[DummyTransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])]\n",
    "        )\n",
    "        # Transformer 模块的堆叠，模型核心部分。\n",
    "\n",
    "        # 使用归一化层（占位符）\n",
    "        self.final_norm = DummyLayerNorm(cfg[\"emb_dim\"])\n",
    "        # 最终归一化层，用于调整特征分布。\n",
    "\n",
    "        self.out_head = nn.Linear(\n",
    "            cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n",
    "        )\n",
    "        # 输出层，将特征映射到词表分布，最终预测输出单词。\n",
    "\n",
    "    def forward(self, in_idx):\n",
    "        batch_size, seq_len = in_idx.shape\n",
    "        # 获取批次大小和序列长度。\n",
    "\n",
    "        tok_embeds = self.tok_emb(in_idx) \n",
    "        # 根据输入索引生成词嵌入。\n",
    "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
    "        # 生成对应的位置信息嵌入。\n",
    "\n",
    "        x = tok_embeds + pos_embeds\n",
    "        # 将词嵌入和位置信息嵌入相加。\n",
    "        x = self.drop_emb(x)\n",
    "        # 应用 Dropout 随机丢弃部分信息。\n",
    "        x = self.trf_blocks(x)\n",
    "        # 通过多个 Transformer 块处理特征。\n",
    "        x = self.final_norm(x)\n",
    "        # 应用最终的归一化层。\n",
    "        logits = self.out_head(x)\n",
    "        # 将隐藏状态映射到词表分布，生成预测结果。\n",
    "        return logits\n",
    "\n",
    "\n",
    "class DummyTransformerBlock(nn.Module):\n",
    "    # Transformer 块的占位类。\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        # 占位，实际模型应实现注意力机制和前馈网络。\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 此块不执行任何操作，仅返回输入。\n",
    "        return x\n",
    "\n",
    "\n",
    "class DummyLayerNorm(nn.Module):\n",
    "    # 归一化层的占位类。\n",
    "    def __init__(self, normalized_shape, eps=1e-5):\n",
    "        super().__init__()\n",
    "        # 参数用于模拟 LayerNorm 的接口。\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 此层不执行任何操作，仅返回输入。\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9665e8ab-20ca-4100-b9b9-50d9bdee33be",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/04.webp?123\" width=\"500px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "794b6b6c-d36f-411e-a7db-8ac566a87fee",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[6109, 3626, 6100,  345],\n",
      "        [6109, 1110, 6622,  257]])\n"
     ]
    }
   ],
   "source": [
    "import tiktoken\n",
    "\n",
    "tokenizer = tiktoken.get_encoding(\"gpt2\")\n",
    "#召唤gpt大神\n",
    "batch = []\n",
    "\n",
    "txt1 = \"Every effort moves you\"\n",
    "txt2 = \"Every day holds a\"\n",
    "\n",
    "batch.append(torch.tensor(tokenizer.encode(txt1)))\n",
    "batch.append(torch.tensor(tokenizer.encode(txt2)))\n",
    "#编码输入文本\n",
    "batch = torch.stack(batch, dim=0)\n",
    "#按照横向来叠加两个向量\n",
    "print(batch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "009238cd-0160-4834-979c-309710986bb0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output shape: torch.Size([2, 4, 50257])\n",
      "tensor([[[-1.2034,  0.3201, -0.7130,  ..., -1.5548, -0.2390, -0.4667],\n",
      "         [-0.1192,  0.4539, -0.4432,  ...,  0.2392,  1.3469,  1.2430],\n",
      "         [ 0.5307,  1.6720, -0.4695,  ...,  1.1966,  0.0111,  0.5835],\n",
      "         [ 0.0139,  1.6754, -0.3388,  ...,  1.1586, -0.0435, -1.0400]],\n",
      "\n",
      "        [[-1.0908,  0.1798, -0.9484,  ..., -1.6047,  0.2439, -0.4530],\n",
      "         [-0.7860,  0.5581, -0.0610,  ...,  0.4835, -0.0077,  1.6621],\n",
      "         [ 0.3567,  1.2698, -0.6398,  ..., -0.0162, -0.1296,  0.3717],\n",
      "         [-0.2407, -0.7349, -0.5102,  ...,  2.0057, -0.3694,  0.1814]]],\n",
      "       grad_fn=<UnsafeViewBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = DummyGPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "logits = model(batch)\n",
    "print(\"Output shape:\", logits.shape)\n",
    "print(logits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8fad0fe-895d-4493-9e48-962e2d46c66f",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "**Note**\n",
    "\n",
    "- 系统为Windows或者Linux, 运行结果如下所示:\n",
    "    \n",
    "```\n",
    "Output shape: torch.Size([2, 4, 50257])\n",
    "tensor([[[-0.9289,  0.2748, -0.7557,  ..., -1.6070,  0.2702, -0.5888],\n",
    "         [-0.4476,  0.1726,  0.5354,  ..., -0.3932,  1.5285,  0.8557],\n",
    "         [ 0.5680,  1.6053, -0.2155,  ...,  1.1624,  0.1380,  0.7425],\n",
    "         [ 0.0447,  2.4787, -0.8843,  ...,  1.3219, -0.0864, -0.5856]],\n",
    "\n",
    "        [[-1.5474, -0.0542, -1.0571,  ..., -1.8061, -0.4494, -0.6747],\n",
    "         [-0.8422,  0.8243, -0.1098,  ..., -0.1434,  0.2079,  1.2046],\n",
    "         [ 0.1355,  1.1858, -0.1453,  ...,  0.0869, -0.1590,  0.1552],\n",
    "         [ 0.1666, -0.8138,  0.2307,  ...,  2.5035, -0.3055, -0.3083]]],\n",
    "       grad_fn=<UnsafeViewBackward0>)\n",
    "```\n",
    "\n",
    "- Since these are just random numbers, this is not a reason for concern, and you can proceed with the remainder of the chapter without issues\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8332a00-98da-4eb4-b882-922776a89917",
   "metadata": {},
   "source": [
    "## 4.2 归一化操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "066cfb81-d59b-4d95-afe3-e43cf095f292",
   "metadata": {},
   "source": [
    "- **层归一化（LayerNorm）**，也称为层标准化（[Ba et al. 2016](https://arxiv.org/abs/1607.06450)），将神经网络层的激活值中心化为均值为 0，并将其方差归一化为 1。\n",
    "- 这种方法能够稳定训练过程，并加速权重的高效收敛。\n",
    "- 在 Transformer 块中，层归一化会在多头注意力模块的前后应用（我们将在后续实现），并在最终输出层之前再次应用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "314ac47a-69cc-4597-beeb-65bed3b5910f",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/05.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "79e1b463-dc3f-44ac-9cdb-9d5b6f64eb9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.2260, 0.3470, 0.0000, 0.2216, 0.0000, 0.0000],\n",
      "        [0.2133, 0.2394, 0.0000, 0.5198, 0.3297, 0.0000]],\n",
      "       grad_fn=<ReluBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "# create 2 training examples with 5 dimensions (features) each\n",
    "batch_example = torch.randn(2, 5) \n",
    "\n",
    "layer = nn.Sequential(nn.Linear(5, 6), nn.ReLU())\n",
    "#一个按照顺序执行的神经网络\n",
    "#具体: 全链接层跟,激活函数\n",
    "out = layer(batch_example)\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fccc29e-71fc-4c16-898c-6137c6ea5d2e",
   "metadata": {},
   "source": [
    "- 计算上述信息的均值与方差"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9888f79e-8e69-44aa-8a19-cd34292adbf5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean:\n",
      " tensor([[0.1324],\n",
      "        [0.2170]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[0.0231],\n",
      "        [0.0398]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "mean = out.mean(dim=-1, keepdim=True)\n",
    "var = out.var(dim=-1, keepdim=True)\n",
    "\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "052eda3e-b395-48c4-acd4-eb8083bab958",
   "metadata": {},
   "source": [
    "- 归一化会单独对两个输入（行）进行处理；\n",
    "- 设置 `dim=-1` 的意思是让计算沿着最后一个维度进行（在这里是特征维度），而不是按行处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "570db83a-205c-4f6f-b219-1f6195dde1a7",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/06.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f8ecbc7-eb14-4fa1-b5d0-7e1ff9694f99",
   "metadata": {},
   "source": [
    "- 通过减去均值并除以方差的平方根（即标准差），可以让输入在列（特征）维度上的均值变为 0，方差变为 1："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9a1d1bb9-3341-4c9a-bc2a-d2489bf89cda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Normalized layer outputs:\n",
      " tensor([[ 0.6159,  1.4126, -0.8719,  0.5872, -0.8719, -0.8719],\n",
      "        [-0.0189,  0.1121, -1.0876,  1.5173,  0.5647, -1.0876]],\n",
      "       grad_fn=<DivBackward0>)\n",
      "Mean:\n",
      " tensor([[-5.9605e-08],\n",
      "        [ 1.9868e-08]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[1.0000],\n",
      "        [1.0000]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "out_norm = (out - mean) / torch.sqrt(var)\n",
    "#执行归一化操作\n",
    "print(\"Normalized layer outputs:\\n\", out_norm)\n",
    "\n",
    "mean = out_norm.mean(dim=-1, keepdim=True)\n",
    "var = out_norm.var(dim=-1, keepdim=True)\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac62b90c-7156-4979-9a79-ce1fb92969c1",
   "metadata": {},
   "source": [
    "- 每个输入的均值都会被调整为 0，方差被归一化为 1。\n",
    "- 为了结果容易阅读，我们可以禁用 PyTorch 的科学计数法："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3e06c34b-c68a-4b36-afbe-b30eda4eca39",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean:\n",
      " tensor([[    -0.0000],\n",
      "        [     0.0000]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[1.0000],\n",
      "        [1.0000]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.set_printoptions(sci_mode=False)\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "944fb958-d4ed-43cc-858d-00052bb6b31a",
   "metadata": {},
   "source": [
    "- 上面我们对每个输入的特征进行了归一化。\n",
    "- 现在，基于相同的思路，我们可以实现一个 `LayerNorm` 类："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3333a305-aa3d-460a-bcce-b80662d464d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "class LayerNorm(nn.Module):\n",
    "    #layer归一化的函数,可以避免信息泄露也可以稳定\n",
    "    def __init__(self, emb_dim):\n",
    "        super().__init__()\n",
    "        self.eps = 1e-5 #避免0的产生导致崩溃\n",
    "        self.scale = nn.Parameter(torch.ones(emb_dim)) #动态的缩放参数\n",
    "        self.shift = nn.Parameter(torch.zeros(emb_dim)) #动态的偏移参数\n",
    "\n",
    "    def forward(self, x):\n",
    "        mean = x.mean(dim=-1, keepdim=True)#算平均值\n",
    "        var = x.var(dim=-1, keepdim=True, unbiased=False)#算方差\n",
    "        norm_x = (x - mean) / torch.sqrt(var + self.eps)#归一化\n",
    "        return self.scale * norm_x + self.shift #通过Ω和  œ 调整归一化后的值范围和位置"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e56c3908-7544-4808-b8cb-5d0a55bcca72",
   "metadata": {},
   "source": [
    "## 缩放与平移\n",
    "\n",
    "- 除了通过减去均值并除以方差来执行归一化操作外，我们还引入了两个可训练参数：`scale`（缩放参数）和 `shift`（平移参数）。\n",
    "- 初始时，`scale` 值为 1，`shift` 值为 0，不会对结果产生影响；但在训练过程中，LLM 会自动调整这两个参数，以提升模型在任务中的表现。\n",
    "- 这种设计使模型能够学习到最适合其数据的缩放和平移方式。\n",
    "- 此外，在计算方差的平方根时，我们会添加一个较小的值（`eps`），以避免方差为 0 时出现除以 0 的错误。\n",
    "\n",
    "## 偏差方差\n",
    "\n",
    "- 在方差计算中，设置 `unbiased=False` 使用公式 $\\frac{\\sum_i (x_i - \\bar{x})^2}{n}$，其中 $n$ 是样本大小（即特征或列的数量）。该公式未使用贝塞尔校正（分母为 `n` 而非 `n-1`），因此提供的是方差的偏差估计。\n",
    "- 对于嵌入维度 $n$ 较大的 LLM 来说，使用 `n` 和 `n-1` 的差异可以忽略不计。\n",
    "- 然而，由于 GPT-2 在归一化层的训练中使用了偏差方差，为了与预训练权重兼容，我们也采用了这种设置。\n",
    "\n",
    "## 实践 LayerNorm\n",
    "\n",
    "- 现在让我们通过实际代码尝试 `LayerNorm` 的应用："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "23b1000a-e613-4b43-bd90-e54deed8d292",
   "metadata": {},
   "outputs": [],
   "source": [
    "ln = LayerNorm(emb_dim=5)#归一化一个五维度\n",
    "out_ln = ln(batch_example)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "94c12de2-1cab-46e0-a099-e2e470353bff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean:\n",
      " tensor([[    -0.0000],\n",
      "        [     0.0000]], grad_fn=<MeanBackward1>)\n",
      "Variance:\n",
      " tensor([[1.0000],\n",
      "        [1.0000]], grad_fn=<VarBackward0>)\n"
     ]
    }
   ],
   "source": [
    "mean = out_ln.mean(dim=-1, keepdim=True)\n",
    "var = out_ln.var(dim=-1, unbiased=False, keepdim=True)\n",
    "\n",
    "print(\"Mean:\\n\", mean)\n",
    "print(\"Variance:\\n\", var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e136cfc4-7c89-492e-b120-758c272bca8c",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/07.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11190e7d-8c29-4115-824a-e03702f9dd54",
   "metadata": {},
   "source": [
    "## 4.3 GELU作为激活函数"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0585dfb-f21e-40e5-973f-2f63ad5cb169",
   "metadata": {},
   "source": [
    "- 在本节中，我们将实现一个小型神经网络子模块，该模块是 LLM 中 Transformer 块的核心组成部分。\n",
    "- 首先，我们从激活函数开始。\n",
    "- 在深度学习中，ReLU（线性整流单元）激活函数因其简单性和在各种神经网络架构中的高效性而被广泛使用。\n",
    "- 在 LLM 中，除了传统的 ReLU，还使用了其他类型的激活函数。其中两个典型的例子是 GELU（高斯误差线性单元）和 SwiGLU（Swish 门控线性单元）。\n",
    "- GELU 和 SwiGLU 是更复杂的平滑激活函数，分别结合了高斯函数和 sigmoid 门控线性单元，提供了比 ReLU 这种简单分段线性函数更好的性能，尤其适用于深度学习模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d482ce7-e493-4bfc-a820-3ea99f564ebc",
   "metadata": {},
   "source": [
    "- **GELU**（[Hendrycks 和 Gimpel, 2016](https://arxiv.org/abs/1606.08415)）可以通过多种方式实现；其精确定义为 $\\text{GELU}(x) = x \\cdot \\Phi(x)$，其中 $\\Phi(x)$ 是标准高斯分布的累积分布函数。\n",
    "- 在实际应用中，通常会使用一种计算成本更低的近似形式：  \n",
    "  $\\text{GELU}(x) \\approx 0.5 \\cdot x \\cdot \\left(1 + \\tanh\\left[\\sqrt{\\frac{2}{\\pi}} \\cdot \\left(x + 0.044715 \\cdot x^3\\right)\\right]\\right)$  \n",
    "  （原始 GPT-2 模型也是使用该近似公式进行训练的）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f84694b7-95f3-4323-b6d6-0a73df278e82",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GELU(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        return 0.5 * x * (1 + torch.tanh(\n",
    "            #这一步把它变得平滑了很多\n",
    "            torch.sqrt(torch.tensor(2.0 / torch.pi)) * \n",
    "            (x + 0.044715 * torch.pow(x, 3))\n",
    "        ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "fc5487d2-2576-4118-80a7-56c4caac2e71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxYAAAEiCAYAAABkykQ1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy80BEi2AAAACXBIWXMAAA9hAAAPYQGoP6dpAABoBElEQVR4nO3deVhUZfsH8O8My7AJiiDIIioqigsqpKG5lYpbRSnZ4p6WhpVLlvgrTXuTytxyt1KSNPelzExcSM0dREWDXEBc2JRVlmGYOb8/kEkElGE7Z4bv57rmet85c5b7nsl5uOc5z/PIBEEQQEREREREVAVysQMgIiIiIiL9x8KCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRlbGwICIiIiKiKmNhQUREREREVcbCgqgMn3/+OWQymSjXDgkJgUwmQ3x8fK1fu7CwEB9//DFcXV0hl8vh7+9f6zFUhJjvERHVbWPGjEHTpk1FubaYbdODBw8wfvx4ODo6QiaTYcqUKaLE8TRivkfEwqJOiouLw+TJk9GqVStYWFjAwsICnp6eCAwMxMWLF0vsW/wPtLxHUlISACA+Ph4ymQzffvttuddt2rQphgwZUuZr586dg0wmQ0hISLXl+TS5ubn4/PPPER4eXmvXfNT8+fOxe/duUa5dnnXr1mHBggUYNmwYfvrpJ0ydOlXUeKT4HhEZsuKivfhhbGwMZ2dnjBkzBnfu3KnUOcPDwyGTybB9+/Zy95HJZJg8eXKZr23fvh0ymaxWv6vv3r2Lzz//HFFRUbV2zWJit03lmT9/PkJCQjBp0iSEhoZi5MiRosUi1feIAGOxA6DatXfvXgwfPhzGxsZ466234OXlBblcjpiYGOzcuROrVq1CXFwc3NzcShy3atUqWFlZlTpf/fr1ayny6pebm4u5c+cCAHr37l3itU8//RQzZ86s0evPnz8fw4YNK9UrMHLkSLz++utQKBQ1ev2yHD58GM7Ozli8eHGtX7ssUnyPiOqCefPmoVmzZsjPz8epU6cQEhKC48ePIzo6GmZmZmKHV+Pu3r2LuXPnomnTpujYsWOJ177//ntoNJoau7bYbVN5Dh8+jGeffRZz5swR5fqPkup7RCws6pTr16/j9ddfh5ubGw4dOoTGjRuXeP3rr7/GypUrIZeX7sgaNmwY7OzsaitU0RkbG8PYWJx/HkZGRjAyMhLl2ikpKXpRLIr5HhHVBQMHDoSPjw8AYPz48bCzs8PXX3+NX3/9Fa+99prI0YnLxMREtGuL2TalpKTA09NTlGvrQsz3iHgrVJ3yzTffICcnB+vXry9VVABF/xg/+OADuLq6ihBdxaSlpeGjjz5C+/btYWVlBWtrawwcOBAXLlwotW9+fj4+//xztGrVCmZmZmjcuDFeffVVXL9+HfHx8bC3twcAzJ07V9vt//nnnwMofY9mu3bt0KdPn1LX0Gg0cHZ2xrBhw7Tbvv32W3Tr1g0NGzaEubk5vL29S90CIJPJkJOTg59++kl77TFjxgAof/zAypUr0bZtWygUCjg5OSEwMBAZGRkl9unduzfatWuHK1euoE+fPrCwsICzszO++eabJ76vxbeyHTlyBJcvX9bGFB4err2N4fEu5+JjHr19bcyYMbCyssKdO3fg7+8PKysr2Nvb46OPPoJarS713i1duhTt27eHmZkZ7O3tMWDAAJw7d06S7xFRXdajRw8ART9QPSomJgbDhg2Dra0tzMzM4OPjg19//VWMEHHz5k2899578PDwgLm5ORo2bIiAgIAyx2JlZGRg6tSpaNq0KRQKBVxcXDBq1Cjcu3cP4eHheOaZZwAAY8eO1X7/FH/XPTrGQqVSwdbWFmPHji11jaysLJiZmeGjjz4CABQUFGD27Nnw9vaGjY0NLC0t0aNHDxw5ckR7jK5tE1A0Nu6LL76Au7s7FAoFmjZtilmzZkGpVJbYr/h25OPHj6NLly4wMzND8+bNsWHDhie+r8VtQFxcHH7//XdtTPHx8eV+F5fVbujy3Vud7XdtvEf0HxYWdcjevXvRokULdO3aVedj09LScO/evRKPx/9gqw03btzA7t27MWTIECxatAgzZszApUuX0KtXL9y9e1e7n1qtxpAhQzB37lx4e3tj4cKF+PDDD5GZmYno6GjY29tj1apVAIBXXnkFoaGhCA0NxauvvlrmdYcPH46jR49qx5QUO378OO7evYvXX39du23p0qXo1KkT5s2bh/nz58PY2BgBAQH4/ffftfuEhoZCoVCgR48e2mu/++675eb9+eefIzAwEE5OTli4cCGGDh2KNWvWoH///lCpVCX2TU9Px4ABA+Dl5YWFCxeidevW+OSTT/DHH3+Ue357e3uEhoaidevWcHFx0cbUpk2bco8pj1qthp+fHxo2bIhvv/0WvXr1wsKFC7F27doS+7399tuYMmUKXF1d8fXXX2PmzJkwMzPDqVOnJPkeEdVlxX84NmjQQLvt8uXLePbZZ/HPP/9g5syZWLhwISwtLeHv749du3bVeoxnz57FiRMn8Prrr+O7777DxIkTcejQIfTu3Ru5ubna/R48eIAePXpg2bJl6N+/P5YuXYqJEyciJiYGt2/fRps2bTBv3jwAwDvvvKP9/unZs2epa5qYmOCVV17B7t27UVBQUOK13bt3Q6lUatuHrKws/PDDD+jduze+/vprfP7550hNTYWfn592LIeubRNQ1KM0e/ZsdO7cGYsXL0avXr0QHBxcol0qdu3aNQwbNgz9+vXDwoUL0aBBA4wZMwaXL18u9/xt2rRBaGgo7Ozs0LFjR21MxX/c66Ii373V3X7XxntEjxCoTsjMzBQACP7+/qVeS09PF1JTU7WP3Nxc7Wtz5swRAJT58PDw0O4XFxcnABAWLFhQbgxubm7C4MGDy3zt7NmzAgBh/fr1T8wjPz9fUKvVJbbFxcUJCoVCmDdvnnbbunXrBADCokWLSp1Do9EIgiAIqampAgBhzpw5pfYpzrtYbGysAEBYtmxZif3ee+89wcrKqsR79uj/FwRBKCgoENq1ayc8//zzJbZbWloKo0ePLnXt9evXCwCEuLg4QRAEISUlRTA1NRX69+9fIvfly5cLAIR169Zpt/Xq1UsAIGzYsEG7TalUCo6OjsLQoUNLXetxvXr1Etq2bVti25EjRwQAwpEjR0psL/7MH/3MRo8eLQAo8VkIgiB06tRJ8Pb21j4/fPiwAED44IMPSsVQ/PkIgjTfIyJDVvxv6+DBg0Jqaqpw69YtYfv27YK9vb2gUCiEW7duafd94YUXhPbt2wv5+fnabRqNRujWrZvQsmVL7bbi75Bt27aVe10AQmBgYJmvbdu2rczvoMc9/t0rCIJw8uTJUv/eZ8+eLQAQdu7cWWr/4u+fJ7VJo0ePFtzc3LTP//zzTwGA8Ntvv5XYb9CgQULz5s21zwsLCwWlUllin/T0dMHBwUEYN26cdpsubVNUVJQAQBg/fnyJ/T766CMBgHD48GHtNjc3NwGAcPToUe22lJQUQaFQCNOnTy91rceV1YY//l1crKx2o6LfvdXdftfme0SCwB6LOiIrKwsAyhyA3bt3b9jb22sfK1asKLXPjh07EBYWVuKxfv36Go/7cQqFQjsGRK1W4/79+7CysoKHhwciIyNLxGtnZ4f333+/1DkqMw1dq1at0LFjR2zZskW7Ta1WY/v27XjxxRdhbm6u3f7o/09PT0dmZiZ69OhRIj5dHDx4EAUFBZgyZUqJ8S8TJkyAtbV1iZ4QoOgzHjFihPa5qakpunTpghs3blTq+pUxceLEEs979OhR4vo7duyATCYrcxBgZT4ffXyPiKSsb9++sLe3h6urK4YNGwZLS0v8+uuvcHFxAVDUi3348GG89tpryM7O1vZk379/H35+frh69WqlZ5GqrEe/e1UqFe7fv48WLVqgfv36pdoHLy8vvPLKK6XOUZnvn+effx52dnYl2of09HSEhYVh+PDh2m1GRkYwNTUFUHQraFpaGgoLC+Hj41Pp9mHfvn0AgGnTppXYPn36dAAo9d3n6empva0NKOoh8fDwqLXvvop891Z3+61v75G+4+iWOqJevXoAirqAH7dmzRpkZ2cjOTm5xD/4R/Xs2bNWBm8/7Uuj+L78lStXIi4ursR9+w0bNtT+/+vXr8PDw6NaB3ANHz4cs2bNwp07d+Ds7Izw8HCkpKSUaDiAolvO/ve//yEqKqrE/ZuVnVf75s2bAAAPD48S201NTdG8eXPt68VcXFxKXatBgwalphKuKcXjJR6/fnp6uvb59evX4eTkBFtb22q5pr69R0RSt2LFCrRq1QqZmZlYt24djh49WmIWtmvXrkEQBHz22Wf47LPPyjxHSkoKnJ2dqy2mp32H5uXlITg4GOvXr8edO3cgCIL2tczMTO3/v379OoYOHVptcRkbG2Po0KHYtGkTlEolFAoFdu7cCZVKVap9+Omnn7Bw4ULExMSUuEWzWbNmlbr2zZs3IZfL0aJFixLbHR0dUb9+/VLffU2aNCl1jse/n2tSRb57q7v91rf3SN+xsKgjbGxs0LhxY0RHR5d6rXjMRU0vNmZmZoa8vLwyXyu+//Vp0xjOnz8fn332GcaNG4cvvvgCtra2kMvlmDJlSo1O/wcUFRZBQUHYtm0bpkyZgq1bt8LGxgYDBgzQ7nPs2DG89NJL6NmzJ1auXInGjRvDxMQE69evx6ZNm2o0vmLlzZb0aCOri/Ia88cHYz/t+lJS3e8RkaHp0qWLdlYof39/PPfcc3jzzTcRGxsLKysr7fftRx99BD8/vzLP8fgfck+iUCiq3D68//77WL9+PaZMmQJfX1/Y2NhAJpPh9ddfr/H24fXXX8eaNWvwxx9/wN/fH1u3bkXr1q3h5eWl3efnn3/GmDFj4O/vjxkzZqBRo0YwMjJCcHBwqUHxuqroD1dSbR9q47tXrPeormFhUYcMHjwYP/zwA86cOYMuXbrU+vXd3Nxw5cqVMl+LjY3V7vMk27dvR58+ffDjjz+W2J6RkVGiR8Xd3R2nT5+GSqUqd2pAXXsQmjVrhi5dumDLli2YPHkydu7cCX9//xK/4u3YsQNmZmb4888/S2wv67axil6/+D2JjY1F8+bNtdsLCgoQFxeHvn376pSHrooHaz4+WP/xX3l04e7ujj///BNpaWlP7LXQl/eIyJAV//Hbp08fLF++HDNnztT+OzMxMamWf19ubm7aduBxurQPo0ePxsKFC7Xb8vPzS313ubu7l/kj26N0bR969uyJxo0bY8uWLXjuuedw+PBh/N///V+p+Jo3b46dO3eWOP/jt4Tqcm03NzdoNBpcvXq1xGQbycnJyMjIeOp7VlU11T5UZ/st9ntU13CMRR3y8ccfw8LCAuPGjUNycnKp12u6Gh80aBBu375daiVlpVKJH374AY0aNULnzp2feA4jI6NScW7btq3UvbxDhw7FvXv3sHz58lLnKD7ewsICQOkvxCcZPnw4Tp06hXXr1uHevXulurmNjIwgk8lK/FoTHx9f5urRlpaWFbp23759YWpqiu+++65E7j/++CMyMzMxePDgCsdfGW5ubjAyMsLRo0dLbF+5cmWlzzl06FAIgqBd4OhRj+aoL+8RkaHr3bs3unTpgiVLliA/Px+NGjVC7969sWbNGiQmJpbaPzU1VafzDxo0CKdOnUJERESJ7RkZGdi4cSM6duwIR0fHJ56jrPZh2bJlpX49Hzp0KC5cuFDmzFXFx1taWmqvXxFyuRzDhg3Db7/9htDQUBQWFpbZPjx6DQA4ffo0Tp48WWI/XdqmQYMGAQCWLFlSYvuiRYsAoMa/+9zd3QGgRPugVqtLzQKoi+puv8V+j+oa9ljUIS1btsSmTZvwxhtvwMPDQ7vytiAIiIuLw6ZNmyCXy7WD8x61ffv2Mgd+9+vXDw4ODtrnhw4dQn5+fqn9/P398c4772DdunUICAjAuHHj0KlTJ9y/fx9btmxBdHQ0NmzYoB3YVp4hQ4Zg3rx5GDt2LLp164ZLly5h48aNJX6lBoBRo0Zhw4YNmDZtGs6cOYMePXogJycHBw8exHvvvYeXX34Z5ubm8PT0xJYtW9CqVSvY2tqiXbt2aNeuXbnXf+211/DRRx/ho48+gq2tbalf6gYPHoxFixZhwIABePPNN5GSkoIVK1agRYsWpe7f9/b2xsGDB7Fo0SI4OTmhWbNmZU4FbG9vj6CgIMydOxcDBgzASy+9hNjYWKxcuRLPPPNMueNiqouNjQ0CAgKwbNkyyGQyuLu7Y+/evUhJSan0Ofv06YORI0fiu+++w9WrVzFgwABoNBocO3YMffr0weTJkwHoz3tEVBfMmDEDAQEBCAkJwcSJE7FixQo899xzaN++PSZMmIDmzZsjOTkZJ0+exO3bt0utL7Rjxw7ExMSUOu/o0aMxc+ZMbNu2DT179sS7776L1q1b4+7duwgJCUFiYmKFJgsZMmQIQkNDYWNjA09PT5w8eRIHDx4sMf6uOI/t27dr2yJvb2+kpaXh119/xerVq+Hl5QV3d3fUr18fq1evRr169WBpaYmuXbs+cSzE8OHDsWzZMsyZMwft27cvNV33kCFDsHPnTrzyyisYPHgw4uLisHr1anh6epYY/6hL2+Tl5YXRo0dj7dq1yMjIQK9evXDmzBn89NNP8Pf3L3P9perUtm1bPPvsswgKCtL2QG/evBmFhYWVPmd1t99iv0d1Ti3PQkUScO3aNWHSpElCixYtBDMzM8Hc3Fxo3bq1MHHiRCEqKqrEvk+abhaPTCVXPPVoeY/Q0FBBEIqm1ps6darQrFkzwcTERLC2thb69Okj/PHHHxWKPT8/X5g+fbrQuHFjwdzcXOjevbtw8uRJoVevXkKvXr1K7Jubmyv83//9n/Zajo6OwrBhw4Tr169r9zlx4oTg7e0tmJqalpi67vHp6h7VvXv3MqeuK/bjjz8KLVu2FBQKhdC6dWth/fr1ZZ4vJiZG6Nmzp2Bubi4A0E6rWt70fcuXLxdat24tmJiYCA4ODsKkSZOE9PT0EvuUNV2sIJSeHrE85R2fmpoqDB06VLCwsBAaNGggvPvuu0J0dHSZ081aWlqWOr6s/AsLC4UFCxYIrVu3FkxNTQV7e3th4MCBQkREhHYfKb5HRIas+N/W2bNnS72mVqsFd3d3wd3dXSgsLBQEQRCuX78ujBo1SnB0dBRMTEwEZ2dnYciQIcL27du1xxVPPVre49ixY4IgCMLt27eF8ePHC87OzoKxsbFga2srDBkyRDh16lSFYk9PTxfGjh0r2NnZCVZWVoKfn58QExMjuLm5lZq2+v79+8LkyZMFZ2dnwdTUVHBxcRFGjx4t3Lt3T7vPnj17BE9PT8HY2LjEd1153xUajUZwdXUVAAj/+9//ynx9/vz5gpubm6BQKIROnToJe/fuLfN8urRNKpVKmDt3rratc3V1FYKCgkpMAywI5U/5Xlb7WZbyjr9+/brQt29fQaFQCA4ODsKsWbOEsLCwMqebreh3b3W337X1HpEgyASBo1GIiIiIiKhqOMaCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRldW5BfI0Gg3u3r2LevXq6bQkPBGRIRMEAdnZ2XBycoJcXnd/c2IbQURUki7tQ50rLO7evQtXV1exwyAikqRbt27BxcVF7DBEwzaCiKhsFWkf6lxhUa9ePQBFb461tbVOx6pUKhw4cAD9+/eHiYlJTYRXKwwhD+YgHYaQhyHkAFQtj6ysLLi6umq/I+uqut5GMAfpMIQ8DCEHwDDyqK32oc4VFsVd29bW1pVqNCwsLGBtba23/2EBhpEHc5AOQ8jDEHIAqiePun77T11vI5iDdBhCHoaQA2AYedRW+1B3b6QlIiIiIqJqw8KCiIiIiIiqTNTCYtWqVejQoYO2y9nX1xd//PHHE4/Ztm0bWrduDTMzM7Rv3x779u2rpWiJiKi2sH0gItI/ohYWLi4u+OqrrxAREYFz587h+eefx8svv4zLly+Xuf+JEyfwxhtv4O2338b58+fh7+8Pf39/REdH13LkRERUk9g+EBHpH1ELixdffBGDBg1Cy5Yt0apVK3z55ZewsrLCqVOnytx/6dKlGDBgAGbMmIE2bdrgiy++QOfOnbF8+fJajpyIiGoS2wciIv0jmVmh1Go1tm3bhpycHPj6+pa5z8mTJzFt2rQS2/z8/LB79+5yz6tUKqFUKrXPs7KyABSNjlepVDrFWLy/rsdJjSHkwRykwxDyMIgc1BrM23sFrdSVy0PKuddU+0BEVFccu3oPh+/KMFAQavQ6ohcWly5dgq+vL/Lz82FlZYVdu3bB09OzzH2TkpLg4OBQYpuDgwOSkpLKPX9wcDDmzp1bavuBAwdgYWFRqZjDwsIqdZzUGEIezEE6DCEPfc5h6w05/k6Wo6HCCDamYTDWsT86Nze3ZgKrgppuHwD++PQ45iAdhpCHIeQA6H8eN9NyMWXrRWTlG8HnbAJe7+Km0/G65C16YeHh4YGoqChkZmZi+/btGD16NP76669yGw9dBQUFlfgVq3iRj/79+1dqjvKwsDD069dPb+cxBgwjD+YgHYaQh77n8PPpBPx9MgYyAK801WCgn+55FP9BLSU13T4A/PGpPMxBOgwhD0PIAdDPPJRqYPElI2Tly+BmJcAi5TL27St7rFp5dPnhSfTCwtTUFC1atAAAeHt74+zZs1i6dCnWrFlTal9HR0ckJyeX2JacnAxHR8dyz69QKKBQKEptNzExqfQfEFU5VkoMIQ/mIB2GkIc+5nDsair+ty8WADC9X0u4PvinUnlIMe+abh8A/vj0OOYgHYaQhyHkAOhvHoIgYMrWi0jMS0ZDS1OMa5Vb4z88iV5YPE6j0ZToln6Ur68vDh06hClTpmi3hYWFlXvPLRGRIbuR+gCBGyOh1gh4tbMz3unRFH/88Y/YYdWYmmgf+ONT2ZiDdBhCHoaQA6B/eaz+6zr2RSfDWC7D8je8kHL5ZI3/8CRqYREUFISBAweiSZMmyM7OxqZNmxAeHo4///wTADBq1Cg4OzsjODgYAPDhhx+iV69eWLhwIQYPHozNmzfj3LlzWLt2rZhpEBHVusxcFcb/dA5Z+YXo3KQ+5r/SHjJoxA6r2rB9ICKqvKP/puKb/TEAgDkvtYWPWwPoeAdUpYhaWKSkpGDUqFFITEyEjY0NOnTogD///BP9+vUDACQkJEAu/28EYrdu3bBp0yZ8+umnmDVrFlq2bIndu3ejXbt2YqVARFTrCtUaTP4lEjfu5cDJxgxrRvrAzMQIKpXhFBZsH4iIKifhfi7e/+U8NAIQ4O2CEV2boLCwsFauLWph8eOPPz7x9fDw8FLbAgICEBAQUEMRERFJ3/9+/wfHrt6DuYkRvh/tA/t6pW/l0XdsH4iIdJdbUIh3Qs8hM08FL9f6+MK/HWQyWa1dX9QF8oiISDebTicg5EQ8AGDxcC+0dbIRNyAiIpIEQRDwyY5LiEnKhp2VKVaP6AwzE6NajYGFBRGRnjh5/T5m74kGAEzv1woD2jUWOSIiIpKKH47F4bcLd2Esl2HlW95obGNe6zGwsCAi0gMJ93MxaWMECjUCXvRywuTnW4gdEhERScTxq/cQ/HBWwM+GeKJLM1tR4mBhQUQkcdn5KozfcBYZuSp0cLHBgmEdavWeWSIikq5babl4/5dIaARgmLcLRvnqtrJ2dWJhQUQkYWqNgCmbo/Bv8gM4WCvw/SifWr9nloiIpCmvQI13QyOQ/vCHp//V8mDtx7GwICKSsAV/xuJQTAoUxnKsHekDB2szsUMiIiIJEAQBQTsv4kpiFhpammL1CG/Rf3hiYUFEJFE7I29j9V/XAQDfDOsAL9f64gZERESSse7veOyOugsjuQwr3uoMp/q1P1j7cSwsiIgk6HxCOmbuvAQACOzjjpc7OoscERERScWJ6/cwf1/RYO1PB7fBs80bihxRERYWREQSk5iZh3dCI1BQqEE/TwdM7+chdkhERCQRt9NzMXnTeag1Al7t7Iwx3ZqKHZIWCwsiIgnJV6nxzoYIpGYr0dqxHpYM7wi5nDNAERFRURsx8ecIpOUUoJ2zNea/0l5SswSysCAikghBEDBj+0VcupMJW0tTfD/KB5YKY7HDIiIiCRAEAbN2XUL0nSzYWppizUjpzRLIwoKISCJWhl9/ZNXUznC1tRA7JCIikoiQE/HYGXkHRnIZlr/ZCc4SGKz9OBYWREQSEHYlGd8eiAUAzH25rWQG4hERkfhO3biP//1eNFh71qA26OZuJ3JEZWNhQUQkstikbEzZfB6CAIzydcNbXcVbNZWIiKTlTkYeAjdGQq0R4N/RCeO6NxU7pHKxsCAiElF6TgHGbziLnAI1fJs3xGdDPMUOiYiIJCJfpcaknyNwP6cAbZ2sEfxqB0kN1n4cCwsiIpGo1Bq8tzESt9Ly4GprjpVvdYaJEb+WiYioaLD2/+2KxsXbmWhgYYLVI7xhbiqtwdqPYwtGRCSS/+29gpM37sPS1Ag/jHoGDSxNxQ6JiIgkIvTUTeyIvA25DFj+pn5M6MHCgohIBL+cScBPJ28CABYP7wgPx3oiR0RERFJx+sZ9zPvtCgAgaGAbdG8hzcHajxO1sAgODsYzzzyDevXqoVGjRvD390dsbOwTjwkJCYFMJivxMDMzq6WIiYiq7mx8GmbviQYAfNS/Ffq3dRQ5IiIikorEzDwEbopEoUbAS15OGN+jmdghVZiohcVff/2FwMBAnDp1CmFhYVCpVOjfvz9ycnKeeJy1tTUSExO1j5s3b9ZSxEREVXMnIw8TQyOgUgsY3KExAvu0EDskIiKSiKKVtSNx70EB2jS2xtdDpT1Y+3GiFhb79+/HmDFj0LZtW3h5eSEkJAQJCQmIiIh44nEymQyOjo7ah4ODQy1FTERUeXkFarwbeg73cwrg2dgaC4bpV4NRm9ijTUR1jSAImL0nGhduZaC+hQnWjpT+YO3HSWqMRWZmJgDA1tb2ifs9ePAAbm5ucHV1xcsvv4zLly/XRnhERJUmCAI+2XER0XeyYGtpirWjvGFhaix2WJLFHm0iqmt+Pp2AreeKBmsve6OTXgzWfpxkWjWNRoMpU6age/fuaNeuXbn7eXh4YN26dejQoQMyMzPx7bffolu3brh8+TJcXFxK7a9UKqFUKrXPs7KyAAAqlQoqlUqnGIv31/U4qTGEPJiDdBhCHrWRw9pjcfj1wl0Yy2X4bngHOFiZVPv1qpKH1D6//fv3l3geEhKCRo0aISIiAj179iz3uOIebSIifXI2Pg1zfy36ofyTAa3Ro6W9yBFVjmQKi8DAQERHR+P48eNP3M/X1xe+vr7a5926dUObNm2wZs0afPHFF6X2Dw4Oxty5c0ttP3DgACwsKlcJhoWFVeo4qTGEPJiDdBhCHjWVw5V0GdbGyAHI4O9WiPv/nMK+f2rkUgAql0dubm4NRFJ9dO3R1mg06Ny5M+bPn4+2bdvWRohERJWSlJmPST8XDdYe3KEx3unZXOyQKk0ShcXkyZOxd+9eHD16tMxehycxMTFBp06dcO3atTJfDwoKwrRp07TPs7Ky4Orqiv79+8Pa2lqna6lUKoSFhaFfv34wMTHR6VgpMYQ8mIN0GEIeNZlD3L0cfLrmNAQUYriPC754qU2NjauoSh7FvblSVFM92gB7tR/HHKTDEPIwhByAms1DWajBxJ/P4d4DJTwcrDD/5TYoLCys9uvUVo+2qIWFIAh4//33sWvXLoSHh6NZM92n01Kr1bh06RIGDRpU5usKhQIKhaLUdhMTk0r/AVGVY6XEEPJgDtJhCHlUdw7Z+SpM2hSF7PxC+Lg1wBf+7WFqXPND2yqTh5Q/u5rq0QbYq10e5iAdhpCHIeQA1Ewem6/LEZUih4WRgNecMhB+8EC1X+NRNd2jLWphERgYiE2bNmHPnj2oV68ekpKSAAA2NjYwNzcHAIwaNQrOzs4IDg4GAMybNw/PPvssWrRogYyMDCxYsAA3b97E+PHjRcuDiOhxGo2AqVuicD01B41tzLBqhHetFBWGpiZ7tAH2aj+OOUiHIeRhCDkANZfH5rO3cfLkFchkwPK3vNGjZc0tgldbPdqiFharVq0CAPTu3bvE9vXr12PMmDEAgISEBMjl/zXG6enpmDBhApKSktCgQQN4e3vjxIkT8PT0rK2wiYieavHBf3HwnxQojOVYM9Ib9vVK95xS+WqjRxtgr3Z5mIN0GEIehpADUL15RNxMw7zfiwbbzfDzwPOejavlvE9T0z3aot8K9TTh4eElni9evBiLFy+uoYiIiKruj0uJWHa46Ffy4Ffbo4NLfXED0kPs0SYiQ5WclY+JP0dCpRYwqL0jJvVyFzukaiOJwdtERIYiJikL07ddAAC8/VwzvNpZt9t3qAh7tInIECkL1Zj0cwRSs5Vo5WCFBcO8DGqhVBYWRETVJCO3AO9siEBugRrd3BsiaGBrsUPSW+zRJiJDNPe3K4hMyIC1mTHWjvSBpcKw/hTnSEIiomqg1gh4/5fzSEjLhUsDcyx/szOMjfgVS0RERX45k4BNpxMgkwFLX++EpnaWYodU7djqERFVgwV/xuLY1XswM5Fj7Ugf2Fqaih0SERFJRGRCOubsKVpZe3q/VujTupHIEdUMFhZERFW09+JdrP7rOgBgwTAveDrpNk0pEREZrpTsfEz6OQIFag0GtHVEYJ8WYodUY1hYEBFVwT+JWZix7SIA4N1ezfGil5PIERERkVQUFGrw3s+RSM5SokUjK3z7mmEN1n4cCwsiokrKyC3Au6ERyFOp0aOlHT7242BtIiL6zxd7r+DczXTUUxhj7UhvWBnYYO3HsbAgIqoEtUbAB5ujkJCWC1dbcyx7oxOM5Ib7KxQREelm69lbCD11s2iw9hsd0dzeSuyQahwLCyKiSlh4IBZH/02FmYkca0b4oL4FB2sTEVGRqFsZ+HR3NABgat9WeL61g8gR1Q4WFkREOvrjUiJWhhcN1v56aAcO1iYiIq3UbCUmhhYN1u7v6YDJBjxY+3EsLIiIdHA1ORsfPVxZe/xzzfByR2eRIyIiIqlQqTUI3BiJpKx8uNtbYuFrXpDXodtkWVgQEVVQVr4K74ZGIOfhytozubI2ERE94svf/8GZ+LSiwdqjfFDPzETskGoVCwsiogrQaARM23IBN+7lwLl+0WBtrqxNRETFdkTcRsiJeADA4uEd4V4HBms/jq0iEVEFLD9yDQf/SYapsRyrRnRGQyuF2CEREZFEXLydgaBdlwAAU/q2RF/PujFY+3EsLIiInuJITAoWH/wXAPA//3bo4FJf3ICIiEgy7j14OFi7UIO+bRzwwfMtxQ5JNCwsiIie4Ob9HHy4+TwEAXiraxO85uMqdkhERCQRxYO172bmo7m9JRYNr1uDtR/HwoKIqBx5BWpM/DkSWfmF6NSkPma/6Cl2SEREJCHz9/2D03FpsFIYY+1IH1jXscHaj2NhQURUBkEQMGvXJfyTmAU7K1OsessbCmMjscMiIiKJ2Bl5G+v/jgcALHzNCy0a1b3B2o9jYUFEVIYNJ29i1/k7MJLLsPzNznC0MRM7JCIikojoO5kI2lk0WPuD51vAr62jyBFJg6iFRXBwMJ555hnUq1cPjRo1gr+/P2JjY5963LZt29C6dWuYmZmhffv22LdvXy1ES0R1RcTNNHyx9woAIGhgazzbvKHIERERkVTcf6DEu6ERUBZq8HzrRpjSt5XYIUmGqIXFX3/9hcDAQJw6dQphYWFQqVTo378/cnJyyj3mxIkTeOONN/D222/j/Pnz8Pf3h7+/P6Kjo2sxciIyVCnZ+XhvYyQKNQIGd2iMt59rJnZIREQkEYVqDSZvOo87GXloZmeJxcM71unB2o8zFvPi+/fvL/E8JCQEjRo1QkREBHr27FnmMUuXLsWAAQMwY8YMAMAXX3yBsLAwLF++HKtXr67xmInIcKkeNhjJWUq0bGSFb4Z2gEzGBoOIiIp89UcMTt64D0tTI6wd6Q0b87o9WPtxohYWj8vMzAQA2NralrvPyZMnMW3atBLb/Pz8sHv37jL3VyqVUCqV2udZWVkAAJVKBZVKpVN8xfvrepzUGEIezEE6DCGP4ti/2R+LM3FpsFQYYdnrXjCVC3qVV1U+C6nlGRwcjJ07dyImJgbm5ubo1q0bvv76a3h4eDzxuG3btuGzzz5DfHw8WrZsia+//hqDBg2qpaiJyJD9eiERPxyPA1A0WLulQz2RI5IeyRQWGo0GU6ZMQffu3dGuXbty90tKSoKDQ8nVDB0cHJCUlFTm/sHBwZg7d26p7QcOHICFhUWlYg0LC6vUcVJjCHkwB+nQ9zzO35ch5N9bAIDhbgWIPfsXnj7iS5oq81nk5ubWQCSVV3yr7DPPPIPCwkLMmjUL/fv3x5UrV2BpaVnmMcW3ygYHB2PIkCHYtGkT/P39ERkZ+cR2hYjoaW7nAMv2XAYATO7TAgPaNRY5ImmSTGERGBiI6OhoHD9+vFrPGxQUVKKHIysrC66urujfvz+sra11OpdKpUJYWBj69esHExP97foyhDyYg3QYQh6xiRn4ePVpAMD455riEz/9HIhXlc+iuDdXKnirLBFJRVpOAX6MNUK+SoPeHvaY2k8/24jaIInCYvLkydi7dy+OHj0KFxeXJ+7r6OiI5OTkEtuSk5Ph6Fj2NF8KhQIKhaLUdhMTk0r/EVSVY6XEEPJgDtKhr3nkKAsxZdtlKDUydGnaADMHtoGxkX7PxF2Zz0Lqn11N3CpLRPQ0hWoNpm69iDSlDE1szbF0eCcYcbB2uUQtLARBwPvvv49du3YhPDwczZo9ffYVX19fHDp0CFOmTNFuCwsLg6+vbw1GSkSGSBAEzNx5CddSc2BtImDJax30vqgwRDV1qyzAcXiPYw7SYQh5GEIOX+2PxYkbaTCVC1j2WjtYmOhnPrU1Bk/UwiIwMBCbNm3Cnj17UK9ePe2Xv42NDczNzQEAo0aNgrOzM4KDgwEAH374IXr16oWFCxdi8ODB2Lx5M86dO4e1a9eKlgcR6aefTsTjtwt3YSyXYWyrQtjXK927SeKrqVtlAY7DKw9zkA5DyENfc4i8J8NPV40AAG+10CD+wknEXxA5qCqq6TF4ohYWq1atAgD07t27xPb169djzJgxAICEhATI5f/9gtitWzds2rQJn376KWbNmoWWLVti9+7dHJhHRDqJTEjHl/v+AQB87NcKDhmXRY6IylKTt8oCHIf3OOYgHYaQhz7n8E9iNj75/jQADcZ3b4L2mht6mUex2hqDJ/qtUE8THh5ealtAQAACAgJqICIiqgvuP1AicGMkVGoBg9s3xhjfJvjjDxYWUlJbt8pyHF7ZmIN0GEIe+pZDek4BAjdHIV+lQc9W9viovwf+3H9D7/IoS02PwZPE4G0iotqi1giYsiUKiZn5aG5via+GtgfXwJMe3ipLRGIoVGvwwebzuJWWhya2Fvju9Y4crK0DjlIkojpl6aGrOHb1HsxNjLB6hDfqmen3r0+GatWqVcjMzETv3r3RuHFj7WPLli3afRISEpCYmKh9Xnyr7Nq1a+Hl5YXt27fzVlki0smCA7HaNmLNSG/UtzAVOyS9Uqkei7i4OBw7dgw3b95Ebm4u7O3t0alTJ/j6+sLMzKy6YyQiqhbhsSlYdvgqAGD+q+3QiqumShZvlSWi2rb34l2s+esGAOCbYR3QprFu46xIx8Ji48aNWLp0Kc6dOwcHBwc4OTnB3NwcaWlpuH79OszMzPDWW2/hk08+gZubW03FTESkszsZeZiyJQqCALzVtQle6fTkgcBERFR3xCRlYca2iwCAd3s2x4teTiJHpJ8qXFh06tQJpqamGDNmDHbs2AFXV9cSryuVSpw8eRKbN2+Gj48PVq5cyV+NiEgSCgo1eG9jJDJyVejgYoPZL3qKHZJBY682EemTjNwCvLMhAnkqNXq0tMPHA1qLHZLeqnBh8dVXX8HPz6/c1xUKBXr37o3evXvjyy+/RHx8fHXER0RUZfP3/YMLtzJgY26CFW92hsLYSOyQDBJ7tYlI36g1Aj7YHIWEtFy42prju9e5snZVVLiweFJR8biGDRuiYcOGlQqIiKg6/X4xESEn4gEAi17zgqtt5RY9oydjrzYR6aOFB2Jx9N9UmJnIsWaEDxpYcrB2VVRqVqiQkJAytxcWFiIoKKgq8RARVZsbqQ/wyY6ie2Yn9XbHC20cRI7IcH311Vc4ffo03nvvvVJFBfBfr/bq1asRExOD5s2bixAlEdF/9l1KxMrw6wCAb4Z5wdOJg7WrqlKFxQcffICAgACkp6drt8XGxqJr16745Zdfqi04IqLKyitQ472NkXigLESXZraY3q+V2CEZNF17tb29vWswGiKiJ4tNysZH2y4AAN7p2RwvcbB2tahUYXH+/Hncvn0b7du3R1hYGFasWIHOnTujdevWuHDhQnXHSESkszm/RiMmKRt2VqZY/kYnGBtx2Z7awl5tIpKyzFwV3gk9h9wCNbq3aIiP/TzEDslgVKqldXd3x99//41XX30VAwYMwNSpU/HDDz9g48aNsLGxqe4YiYh0su3cLWw9dxtyGfDd653QyJozEdUm9moTkVSpNQI+3HIeN+/nwrm+OZa90Zk/PFWjSr+Tv//+OzZv3gxfX1/Ur18fP/74I+7evVudsRER6Sw2KRuf7YkGAEzt2wrdWtiJHFHdw15tIpKqxWH/Ijz24WDtkd6w5WDtalWpwuLdd99FQEAAPvnkExw7dgwXL16Eqakp2rdvj61bt1Z3jEREFZKjLMSkjRHIV2nQs5U9Avu0EDukOom92kQkRfujE7H8yDUAwFevdkA7Z34fVbdKFRZ///03Tp8+jenTp0Mmk8HR0RH79u3DvHnzMG7cuOqOkYjoqQRBwKxdl3AjNQeO1mZYMrwj5JyLXDTs1SYiKbmanI3pW4t6TMd1bwb/Ts4iR2SYKlVYREREwMvLq9T2wMBAREREVDkoIiJd/XLmFvZE3YWRXIblb3Zi97aI2KtNRFKSmafCO6ERyClQ49nmtpg1iCtr15QKL5D3KIVCUe5rHh4cWU9EtSv6TiY+/+0yAOBjPw/4NLUVOaK6rbhXu/gHqOJe7RUrVmDcuHF47bXXRI6QiOoKjUbA1C1RiLuXAycbM6x4k4O1a1KF39kBAwbg1KlTT90vOzsbX3/9NVasWFGlwIiIKiI7X4XJmyJRUKjBC60bYUIPLrwmNvZqE5FULDl0FYdjUmBqLMeakT5oaFX+j+NUdRXusQgICMDQoUNhY2ODF198ET4+PnBycoKZmRnS09Nx5coVHD9+HPv27cPgwYOxYMGCmoybiAiCIGDmzkuIfzht4MLXvDiuQgLYq01EUvDn5SR8d+gqACD4lfZo78LB2jWtwj0Wb7/9Nm7cuIFZs2bhypUreOedd9CjRw8888wz8PPzw/fff48mTZrg7Nmz2LJlC5o0afLUcx49ehQvvvginJycIJPJsHv37ifuHx4eDplMVuqRlJRU0TSIyID8fOomfr+YCGO5DMve7IT6FhxXIRb2ahORlFxL+W+w9phuTTHU20XkiOoGncZYKBQKjBgxAiNGjAAAZGZmIi8vDw0bNoSJiYnOF8/JyYGXlxfGjRuHV199tcLHxcbGwtraWvu8UaNGOl+biPTbpduZ+GLvPwCAmQNbo3OTBiJHVLexV5uIpCIrv2iw9gNlIbo2s8X/DW4jdkh1RqUGbxezsbGp0pzkAwcOxMCBA3U+rlGjRqhfv36lr0tE+i0rX4XATZEoUGvQz9MBbz/XTOyQ6ry3334bI0aMwLZt27BlyxasXbsWmZmZAACZTAZPT0/4+fnh7NmzaNOGjTwR1QyNRsC0LVG4kZqDxjZmWPFWZ5hwsHat0amw+O6778rcbmNjg1atWsHX17dagnqajh07QqlUol27dvj888/RvXv3cvdVKpVQKpXa51lZWQAAlUoFlUql03WL99f1OKkxhDyYg3TUdh6CIODjbReRkJYL5/pmCPb3RGFhYZXOyc+ienKv7l5tIiJdfXf4Kg7+UzRYe/UIb9hxsHat0qmwWLx4cZnbMzIykJmZiW7duuHXX3+FrW3NTPXYuHFjrF69Gj4+PlAqlfjhhx/Qu3dvnD59Gp07dy7zmODgYMydO7fU9gMHDsDCwqJScYSFhVXqOKkxhDyYg3TUVh7HkmTYH2cEI5mA4S4P8PeR6rtuXf4scnNzqz2OqvZqExHp4uCVZCw5WDRY+0v/dvByrS9uQHWQToVFXFxcua/duHEDI0aMwKeffoqVK1dWObCyeHh4lJhRpFu3brh+/ToWL16M0NDQMo8JCgrCtGnTtM+zsrLg6uqK/v37lxinUREqlQphYWHo16+fXv/6Zgh5MAfpqM08Lt/NwkdrTwMQ8MmA1hjbza1azsvP4r/e3Kqo7l7to0ePYsGCBYiIiEBiYiJ27doFf3//cvcPDw9Hnz59Sm1PTEyEo6OjTtcmIv1yPfUBpm6JAgCM8nVDgI+ruAHVUVUaY/Go5s2b46uvvsK4ceOq65QV0qVLFxw/frzc1xUKRZlTH5qYmFT6D4iqHCslhpAHc5COms4jK1+FD7dehEotoG8bB0zo6Q6ZrHqnlq3Ln0V15F3dvdqc4IOIKiI7X4V3NpxDtrIQXZra4rMhnmKHVGdVW2EBAE2aNKn1qV+joqLQuHHjWr0mEdUuQRAQtOMSbj5cr+LbgA7VXlRQ1VV3rzYn+CCip9FoBEzfegHXU3PgaM3B2mKr1sLi0qVLcHOr+K0JDx48wLVr17TP4+LiEBUVBVtbWzRp0gRBQUG4c+cONmzYAABYsmQJmjVrhrZt2yI/Px8//PADDh8+jAMHDlRnGkQkMT+fTsDvl4rWq1jO9Sr0Um32ausywQcR6bcVR67hwJVkmBrJsWpEZ9jX42BtMelUWJR3D25mZiYiIiIwffp0jB49usLnO3fuXIn7YYvHQowePRohISFITExEQkKC9vWCggJMnz4dd+7cgYWFBTp06ICDBw+WeU8tERmG6DuZ+OK3KwCATwa0RieuV6G3arpXuzITfHDmwJKYg3QYQh41ncOR2FQsOvgvAODzF9ugXWOrGrlWXf8sdDlGp8Kifv365d5+IJPJMH78eMycObPC5+vduzcEQSj39ZCQkBLPP/74Y3z88ccVPj8R6bfsfBUmP1yv4oXWjTC+B9er0Ge69mrrqjITfHDmwLIxB+kwhDxqIoeUPGDRJSMIggzdHTSwTL6AffsuVPt1HlVXPwtdZg3UqbA4cuRImdutra3RsmVLmJmZISUlBU5OTrqcloioFEEQMGtXNOLv58LJxgzfBnhxXIXEVXevdnV42gQfnDmwJOYgHYaQR03l8EBZiIA1p5GnzoF3k/pYO9YHpsY1N66irn8WuswaqFNh0atXrye+fuHCBXTu3BlqtVqX0xIRlfLLmVv47cJdGMllWPZmJzSw5LgKqavuXu3q8LQJPjhzYNmYg3QYQh7VmYMgCAjafBHXUnPgYK3AqpHesDSvnXEVdfWz0GX/ah28TURUHf5JzMLc3y4DAGb4ecDbrWYW3aTqVd292pzgg4getzL8OvZfToKJkQyrRnijUT0zsUOiR7CwICJJyVEWInBTJJSFGvT2sMc7PZqLHRJVUHX3anOCDyJ61JHYFHx7IBYAMO/ldujMyTwkh4UFEUmGIAj4dHc0bjycj3zRax0hl3NcRV3FCT6IqFj8vRx8+Mt5CALwZtcmeKNLE7FDojLoVFhcvHjxia/HxsZWKRgiqtu2nbuNXefvwEguw3dvdIItx1UQEdV5OcpCvBsagaz8QnRuUh9zXuTK2lKlU2HRsWNHyGSyMn9BKt7OWVuIqDL+Tc7G7F+jAQDT+rVCl2YcV0FEVNcJgoAZ2y8gNjkb9vUUWDXCGwpjI7HDonLoVFjExcXVVBxEVIflFhQicGMk8lUa9Ghph0m93MUOiSqBvdpEVN1W/3UD+y49HKz9Vmc4WHOwtpTpVFjU5MJGRFR3zdlzGVdTHqBRPQUWD+e4Cn3FXm0iqk5//ZuKb/6MAQB8/lJb+DRlT7bU6VRYfPPNN3j//fdhbm4OAPj777/h4+OjnQM8Ozsbn3zyCVauXFn9kRKRQdoRcRvbIm5DLgOWvt4Jdla1Mx85VT/2ahNRdbl5PwcfPBys/fozrniTg7X1gk6FRVBQEMaMGaMtLAYOHIioqCg0b140HWRubi7WrFnDwoKIKuRaSjY+3V00rmJK31bwdW8ockRUFezVJqLqkFtQNFg7M0+Fjq71Mffltuzt1BM6rX/+ePf2k6YBJCJ6krwCNQI3nkeeSo3uLRoisE8LsUOianTs2DGMGDECvr6+uHPnDgAgNDQUx48fFzkyIpIyQRDw8faLiEnKhp2VKVaN6MzB2npEp8KCiKi6fP7rZcQmZ8POSoElwzvBiOMqDMaOHTvg5+cHc3NznD9/HkqlEgCQmZmJ+fPnixwdEUnZ98duYO/FRBjLZVj5ljca25iLHRLpgIUFEdW6nZG3seXcLchkwHevd4R9PY6rMCT/+9//sHr1anz//fcwMTHRbu/evTsiIyNFjIyIpOz41Xv46o+iwdpzXvTktON6SOeVt3/44QdYWVkBAAoLCxESEgI7OzsARYO3iYie5FpKNv5vV9G4ig9faIluLexEjoiqW2xsLHr27Flqu42NDTIyMmo/ICKSvFtpuZj8SyQ0AhDg7YIRz3LMlj7SqbBo0qQJvv/+e+1zR0dHhIaGltqHiKgsj46r6ObeEO8/31LskKgGODo64tq1a2jatGmJ7cePH9dO9kFEVCyvQI13QiOQkauCl4sNvvBvx8HaekqnwiI+Pr6GwiCiumDOr9H/jat4vSPHVRioCRMm4MMPP8S6desgk8lw9+5dnDx5EtOnT8fs2bPFDo+IJEQQBMzceRH/JGY9HKztDTMTDtbWVzoVFvn5+Th48CCGDBkCoGj62eJBeQBgbGyMefPmwcyMqyISUUk7Im5j67mi9Sq+e70jGtXj94ShmjlzJjQaDV544QXk5uaiZ8+eUCgUmDFjBsaPHy92eEQkIT8ej8OeqLswlsuw4s3OcKrPwdr6TKfB2yEhIVizZo32+fLly3HixAmcP38e58+fR2hoqE5rWBw9ehQvvvginJycIJPJsHv37qceEx4ejs6dO0OhUKBFixYICQnRJQUiEsHV5P/Wq/jwhVYcV2HgZDIZ/u///g9paWmIjo7GqVOnkJqaChsbGzRr1kzs8IhIIk5cu4f5+/4BAHw6uA26NudaRvpOp8Ji48aNeOedd0ps27RpE44cOYIjR45gwYIF2LZtW4XPl5OTAy8vL6xYsaJC+8fFxWHw4MHo06cPoqKiMGXKFIwfPx5//vmnLmkQUS3KLSjEexsjkadS47kWdpj8PNerMFRKpRJBQUHw8fFB9+7dsW/fPnh6euLy5cvw8PDA0qVLMXXqVLHDJCIJuJWWi8BNRYO1h3Z2wehuTcUOiaqBTrdCXbt2De3bt9c+NzMzg1z+X23SpUsXBAYGVvh8AwcOxMCBAyu8/+rVq9GsWTMsXLgQANCmTRscP34cixcvhp+fX4XPQ0S1QxAEfLo7GldTHsC+ngKLh3NchSGbPXs21qxZg759++LEiRMICAjA2LFjcerUKSxcuBABAQEwMuK900R1XV6BGu+GRiA9V4X2zjb48hUO1jYUOhUWGRkZJcZUpKamlnhdo9GUeL26nTx5En379i2xzc/PD1OmTKmxaxJR5W07dxs7I+9ALgOWvdGJ61UYuG3btmHDhg146aWXEB0djQ4dOqCwsBAXLlzgHw1EBKDoB6egnRdxJTELDS1NsXokB2sbEp0KCxcXF0RHR8PDw6PM1y9evAgXF5dqCawsSUlJcHBwKLHNwcEBWVlZyMvLg7l56QE/SqWyRLGTlZUFAFCpVFCpVDpdv3h/XY+TGkPIgzlIR3l5xCRl47M9ReMqpr7QAt6u1pLN1dA/C12OrYrbt2/D29sbANCuXTsoFApMnTqVRQURaa37Ox67o+7CSC7D8jc7w5mDtQ2KToXFoEGDMHv2bAwePLjUzE95eXmYO3cuBg8eXK0BVlVwcDDmzp1bavuBAwdgYWFRqXOGhYVVNSxJMIQ8mIN0PJpHvhpYeNEIykIZ2tTXwOVBDPbtixExuooxxM+ionJzc6t8XbVaDVNTU+1zY2Nj7YKqREQnrv83WPv/BrWBrzsHaxsanQqLWbNmYevWrfDw8MDkyZPRqlUrAEWrrC5fvhyFhYWYNWtWjQQKFC26lJycXGJbcnIyrK2ty+ytAIqmxJ02bZr2eVZWFlxdXdG/f39YW1vrdH2VSoWwsDD069cPJiYmuicgEYaQB3OQjsfzEAQBU7ZeREp+MhytFfhpki8aWJg+/UQiMtTPQhfFvblVIQgCxowZA4Wi6Ja3/Px8TJw4EZaWliX227lzZ5WvRUT65U5GHiZvOg+1RsArnZwxtntTsUOiGqBTYeHg4IATJ05g0qRJmDlzJgRBAFA0tWC/fv2wcuXKUrcqVSdfX1/s27evxLawsDD4+vqWe4xCodA2co8yMTGp9B8QVTlWSgwhD+YgHcV5hPwdh33RyTCWy7ByhDca2Vg+/WCJMLTPQtdjqmr06NElno8YMaJK5zt69CgWLFiAiIgIJCYmYteuXfD393/iMeHh4Zg2bRouX74MV1dXfPrppxgzZkyV4iCiqslXqfFu6Dmk5RSgrZM1gl9tz1skDZROhQUANGvWDPv370daWhquXbsGAGjRogVsbW11vviDBw+05wCKppONioqCra0tmjRpgqCgINy5cwcbNmwAAEycOBHLly/Hxx9/jHHjxuHw4cPYunUrfv/9d52vTUTVLzIhHV8+7OaeNagNOjdpIHJEVJvWr19frecrnpJ83LhxePXVV5+6f/GU5BMnTsTGjRtx6NAhjB8/Ho0bN+bMgUQiEQRg9q9XEH0nCw0sTLCGg7UNms6FRTFbW1t06dKlShc/d+4c+vTpo31efMvS6NGjERISgsTERCQkJGhfb9asGX7//XdMnToVS5cuhYuLC3744Qc2GEQSkJZTgMkbI6FSCxjU3pHd3FRlnJKcSP8dS5JhV3wi5DJgxZud4dKgcuNbST9UurCoDr1799beTlWWslbV7t27N86fP1+DURGRrjQCMH37JdzNzEczO0t8PbQDu7mp1lVmSnLOHFgSc5AOQ8jj5LVU7IovWu/sE79WeMbNRi/zMYTPorZmDRS1sCAiw/DnbTmO374PMxM5Vo3ojHpm+j9OgfRPZaYk58yBZWMO0qGveaQrgW8vGkEDGbztNHDIuIJ9+66IHVaV6Otn8aianjWQhQURVcnRq/fw5+2i3ongV9ujtaNus60RiYkzB5bEHKRDn/NQqtR448ezeFCYBWcLAWsn9Ia1hdnTD5Qoff4sitXWrIEsLIio0m6n52L6tksQIMObXVzwSqeaWyCT6GkqMyU5Zw4sG3OQDn3LQxAEBO2+gkt3slDf3ARve+TB2sJMr3Ioj759FmWp6VkD5boGREQEFE0fOOnnSGTkqdDEUsCsga3FDonqOF9fXxw6dKjEtqdNSU5E1Sv01E1sj7gNuQxYMrwDGupvRwVVAgsLItKZIAiYvScal+5kooGFCcZ6qKEw5tcJVa8HDx4gKioKUVFRAP6bkrx4tsCgoCCMGjVKu//EiRNx48YNfPzxx4iJicHKlSuxdetWTJ06VYzwieqcM3FpmPdb0TiKmQNboztX1q5z+JcAEels89lb2Hru4S9Sr3WAbek7SYiq7Ny5c+jUqRM6deoEoGhK8k6dOmH27NkAUO6U5GFhYfDy8sLChQs5JTlRLUnMzMN7GyNQqBHwopcTJvRoLnZIJAKOsSAinZxPSMecPZcBAB/5eaCbe0PsixU5KDJInJKcSD8oC9WY+HMk7j0oQGvHevh6KFfWrqvYY0FEFZaSnY9JP0eiQK2BX1sHTOrlLnZIREQkIkEQMGfPZVy4lQEbcxOsHekDC1P+bl1XsbAgogopKNQgcGMkkrLy4W5viW8DvPiLFBFRHbfxdAI2n70FuQxY9kYnNGnIlbXrMhYWRFQhX/5+BWfj02GlMMbaUT5cBI+IqI47F5+Gub8V3Rr78YDW6NnKXuSISGwsLIjoqbaeu4WfTt4EACwe3hHu9lYiR0RERGJKzsrHpI2RUKkFDG7fGO/25GBtYmFBRE8RmZCOT3dFAwA+fKEl+nk6iBwRERGJqWiwdgRSs5XwcKiHb4Z14K2xBICFBRE9QXJWPiaGRqBArUF/Twd8+EJLsUMiIiKRff7rFZxPyIC1mTHWjPSGpYKDtakICwsiKlO+So13QiOQkq1EKwcrLBreEXI5f5EiIqrLNp1OwC9nEiCTAd+90QlN7SzFDokkhIUFEZUiCAKCdl7STh/4/SgfWPEXKSKiOi3iZjrm/Fp0a+xH/T3Q26ORyBGR1LCwIKJSVv11HbvO34GRXIaVb3WGW0P+IkVEVJelZOVj0s8RUKkFDGrviPd6cx0jKo2FBRGVcOByEhb8WbSU9ucveqJ7CzuRIyIiIjEVFGowaWOk9tbYBcO4jhGVjYUFEWlduZuFKVuiIAjAiGebYKRvU7FDIiIikc3bexkRN9NhbWaMtSN9OFibysXCgogAFM0A9fZPZ5FboEY394aY82JbsUMiIiKRbT17Cz+fKhqsvfR1DtamJ5NEYbFixQo0bdoUZmZm6Nq1K86cOVPuviEhIZDJZCUeZmZmtRgtkeHJLSjE+J/OITEzH+72llj1ljdMjCTx9UBERCI5n5COT3cXDdae3q8V+rTmYG16MtH/ctiyZQumTZuGOXPmIDIyEl5eXvDz80NKSkq5x1hbWyMxMVH7uHnzZi1GTGRYNBoBU7dE4dKdTNhammLdmGdgY2EidlhERCSilOx8TPo5EgVqDQa0dURgnxZih0R6QPTCYtGiRZgwYQLGjh0LT09PrF69GhYWFli3bl25x8hkMjg6OmofDg5cCZiosr7c9w/+vJwMUyM51o705gxQRER1XEGhBoEbI5GUlY8Wjazw7WscrE0VI+rom4KCAkRERCAoKEi7TS6Xo2/fvjh58mS5xz148ABubm7QaDTo3Lkz5s+fj7Zty74fXKlUQqlUap9nZWUBAFQqFVQqlU7xFu+v63FSYwh5MIfqEXLyJn48HgcA+OrVtvByrlcn/10YQg5A1fLQ99yJqPr87/crOBufjnoKY6wd6c11jKjCRP0v5d69e1Cr1aV6HBwcHBATE1PmMR4eHli3bh06dOiAzMxMfPvtt+jWrRsuX74MFxeXUvsHBwdj7ty5pbYfOHAAFhYWlYo7LCysUsdJjSHkwRwq78J9Gdb/Kwcgw0tN1DC6fR77bp+v9Pn4WUhHZfLIzc2tgUiISN9sPXcLG04W3WK+eHhHNLe3Ejki0id6V4L6+vrC19dX+7xbt25o06YN1qxZgy+++KLU/kFBQZg2bZr2eVZWFlxdXdG/f39YW1vrdG2VSoWwsDD069cPJib6ew+6IeTBHKrm3M10bAyJgAAN3uzigs+HtKl0Nzc/C+moSh7FvblEVHdF3crAp7uKBmtP7dsKfT15qznpRtTCws7ODkZGRkhOTi6xPTk5GY6OjhU6h4mJCTp16oRr166V+bpCoYBCoSjzuMr+AVGVY6XEEPJgDrqLTcrGuz+fh7JQg75tGmHey+1hXA0zQPGzkI7K5GEIeRNR5aVmKzExNAIFag36eTrg/ec5WJt0J+rgbVNTU3h7e+PQoUPabRqNBocOHSrRK/EkarUaly5dQuPGjWsqTCKDcTs9F6PWnUZWfiG83Rpg2Rudq6WoICIi/aVSaxC4qWiwdnN7Syx6zQtyOQdrk+5E/4ti2rRp+P777/HTTz/hn3/+waRJk5CTk4OxY8cCAEaNGlVicPe8efNw4MAB3LhxA5GRkRgxYgRu3ryJ8ePHi5UCkV64/0CJUevOIDlLiZaNrPDjaB+YmxqJHRbRE3GdI6Ka9+Xv/+BMXBqsFEUra9czYw8mVY7oYyyGDx+O1NRUzJ49G0lJSejYsSP279+vHdCdkJAAufy/+ic9PR0TJkxAUlISGjRoAG9vb5w4cQKenp5ipUAkeVn5KoxadwY3UnPgZGOGDW93QX0LU7HDInqi4nWOVq9eja5du2LJkiXw8/NDbGwsGjUqe6Eua2trxMbGap9zikyiJ9sRcRshJ+IBFA3WbtGIg7Wp8kQvLABg8uTJmDx5cpmvhYeHl3i+ePFiLF68uBaiIjIMeQVqvB1yFpfvZqGhpSlCx3dFYxtzscMieqpH1zkCgNWrV+P333/HunXrMHPmzDKPKV7niIie7tLtTATtugQA+PCFlujHwdpURZIoLIioZigL1Xj354ii+cjNjLHh7S5w59SBpAdqY50jgGsdPY45SEdN53E/pwDvhJ5DQaEGz3vY472eTav9WvwspKO21jliYUFkoAoKNXjv50gc/TcV5iZGCBn7DNo62YgdFlGF1MY6RwDXOioPc5COmshDrQFW/iNHYpYcjcwE9LdOxP79idV+nWL8LKSjptc5YmFBZIBUag0mb4rEoZgUKIzl+HG0D7zdbMUOi6hG6brOEcC1jh7HHKSjJvP4cl8MrmUlwNLUCD9N6Fpj4yr4WUhHba1zxMKCyMCo1Bp8uPk8DlxJhqmxHN+P8kG3FnZih0Wkk9pY5wjgWkflYQ7SUd157Dp/GyEnEwAAC1/riDbODart3OXhZyEdNb3OkejTzRJR9SkoLOqp2HcpCaZGcqwZ6Y2erezFDotIZ1zniKj6Rd/JxMwdRYO1J/dpgQHtONEBVS/2WBAZiHyVGu9tjMThmBSYGsuxekRn9PEoe0pOIn0wbdo0jB49Gj4+PujSpQuWLFlSap0jZ2dnBAcHAyha5+jZZ59FixYtkJGRgQULFnCdI6KH0nIK8G5oBJSFGvTxsMfUfq3EDokMEAsLIgOQW1CId0MjcOzqPZiZyLF2pA97KkjvcZ0joupR+HDc3Z2MPDRtaIElr3eCEVfWphrAwoJIz2XkFmBcyFlEJmTAwtQIP45+Br7uDcUOi6hacJ0joqr7en8MTly/D0tTI6wd5QMbc/0eJ0DSxcKCSI8lZ+Vj1I9nEJucDWszY6wf+wxnfyIiIq09UXfw/bE4AMC3AV5o5VBP5IjIkLGwINJT11MfYMz6M7iVlodG9RQIfbsrPBzZYBARUZHLdzPxyY6LAIDAPu4Y2J4TGVDNYmFBpIfOxqdhwoZzyMhVwa2hBX5+uytcbSu3mBcRERme9IeDtfNVGvT2sMe0fh5ih0R1AAsLIj2z9+JdTNt6AQWFGnR0rY8fRvvAzqr0PPxERFQ3Fao1eP+X87idnge3hhZYOpyDtal2sLAg0hMajYClh65i6aGrAAC/tg5YMrwTzE2NRI6MiIik5Js/Y3H82j1YmBphzUhv2FhwsDbVDhYWRHogR1mI6VsvYP/lJADAuO7N8H+D2/AXKCIiKuHXC3ex9ugNAMCCYV5o7WgtckRUl7CwIJK4+Hs5mPhzBGKSsmFiJMOX/u3x2jOuYodFREQS809iFj7efgEAMLGXOwZ34GBtql0sLIgkbH90ImZsu4hsZSHsrBRYM7Izp5MlIqJS0nMK8E7oOeSrNOjR0g4z/DhYm2ofCwsiCVIWqvHN/lj8eLxo7vFnmjbAsjc6w9HGTOTIiIhIatQaAR9sPo9baXlwtTXHsjc4WJvEwcKCSGKupWTjg1+icCUxCwDwTs/mmOHnARMjuciRERGRFC34MxbHrt6DuYkR1o70QX0LU7FDojpKEn+prFixAk2bNoWZmRm6du2KM2fOPHH/bdu2oXXr1jAzM0P79u2xb9++WoqUqOZoNAI2nIzH4O+O40piFhpYmGDtSG/MGtSGRQUREZVp78W7WP3XdQDA18M6oE1jDtYm8Yj+18qWLVswbdo0zJkzB5GRkfDy8oKfnx9SUlLK3P/EiRN444038Pbbb+P8+fPw9/eHv78/oqOjazlyouoTfy8Hb3x/CrP3XIaysOj+2D+n9ET/to5ih0ZERBIVk5SFGduKVtZ+p2dzvOTlJHJEVNeJXlgsWrQIEyZMwNixY+Hp6YnVq1fDwsIC69atK3P/pUuXYsCAAZgxYwbatGmDL774Ap07d8by5ctrOXKiqlNrgB+Ox2PA0qM4HZcGcxMjzHnREz+N7YJG1hxPQUREZcvILcA7GyKQp1LjuRZ2+JiDtUkCRB1jUVBQgIiICAQFBWm3yeVy9O3bFydPnizzmJMnT2LatGkltvn5+WH37t1l7q9UKqFUKrXPs7KK7ltXqVRQqVQ6xbsj4hYupciQH3kLChMTGMllMJbLYGwkg5FcBlMjOYzlMpgYyR8+ZDAxlsPUSA5TYzkUDx/GchlkMvEGVRXnrWv+UmIIORz7NwXfXDRCUt6/AIBuzW3xxcueaGJrAbW6EGq1yAFWkCF8FoaQA1C1PPQ9d6K6RK0R8OHmKCSk5cKlQdFgbWPeMksSIGphce/ePajVajg4OJTY7uDggJiYmDKPSUpKKnP/pKSkMvcPDg7G3LlzS20/cOAALCwsdIp37hkj5KmNsPH6Pzod9zgZBJjIoX2YygFTo4f/KxegMELRQw4ojAEzIwFmRoCZEWBuBJgbCzA3AiyMAXPjouMqU6eEhYVVKQ8p0MccUvOAvbfkiLovByCDpbGAl9w06GqfguhTKdDXm/r08bN4nCHkAFQuj9zc3BqIhIhqwsIDsfjr31SYmcixZqQ3GlhysDZJg8HPChUUFFSihyMrKwuurq7o378/rK11G+C0L/M8Eu4mo36DhhAAFGoEFGoEqDUCVGoBhWoNVGoBKrUGhZqi/y0o1KDg4fZiAmQo0AAFmrKuonuFYGosR31zE9Q3N0EDSxPYWpjC1tIUDS1NYWtlCjtLU9jXU8DOyhSN6ilgBA3CwsLQr18/mJiY6Hw9KVCpVHqXw70HSiw/cgNbLt5GoUaAXAZ0d9Dgm5E9YWetW5ErJfr4WTzOEHIAqpZHcW8uEUnbvkuJWBn+cLD20A5o62QjckRE/xG1sLCzs4ORkRGSk5NLbE9OToajY9mDVh0dHXXaX6FQQKFQlNpuYmKic8O7/I1O2LdvHwYNekbnYzUaAQVqDZQqDZSFauSrNMgvVCNfpUZegRq5KjXyC9TIKVAjr6AQOQVq5CgL8UBZiBxlIbLzix8qZOcXIjNPhcw8FQo1AgoKNUjJViIlW/n0QABYmxnDQmaEbakX4VTfHI425nCyMYNTfXM41TeHc31zmJsa6ZSfWCrzOda2xMw8fH80Dr+cSUCequj+pl6t7DG9bwvEnT8GO2sLyedQEfrwWTyNIeQAVC4PQ8ibyND9m5yNj7YVraw9/rlmeLmjs8gREZUkamFhamoKb29vHDp0CP7+/gAAjUaDQ4cOYfLkyWUe4+vri0OHDmHKlCnabWFhYfD19a2FiCtPLpfBTG4EMxMjANXTgAuCgNwCNdJzC5CRq0J6bgHScv573HtQgHsPlLj/QInUB0qkZCmhLNQgK78QWZAh6dr9cs9tZ2UK5wYWcGlgjia2FnBtYIEmthZwa2gBp/rmXHinAv5JzELI3/HYef62tseqo2t9fDKgNXzdG0KlUiHuvMhBEhGRXsjMU+GdDeeQW6BGN/eGmDmwtdghEZUi+q1Q06ZNw+jRo+Hj44MuXbpgyZIlyMnJwdixYwEAo0aNgrOzM4KDgwEAH374IXr16oWFCxdi8ODB2Lx5M86dO4e1a9eKmYYoZDIZLBXGsFQYw6XB0/cXBAHZykLcuf8Avx08Brc2HZD6QIW7mflIyszH3Yw83EnPQ7ay8GFRUoALtzJKncfESAbXBkVFRlM7SzR75OFkYw55HS468lVqhF1JRuipmzgTl6bd3rWZLSY/3wLPtbATdeA+ERHpH41GwNQtUYi/nwvn+uZY/mZnDtYmSRK9sBg+fDhSU1Mxe/ZsJCUloWPHjti/f792gHZCQgLk8v/+8XTr1g2bNm3Cp59+ilmzZqFly5bYvXs32rVrJ1YKekMmk8HazATmjazgUV/AoE7OZd7+kJmnwu30XNxKy3v4v7m4mZaLhLRc3E7LQ4Fagxv3cnDjXg4Qm1riWFNjOZo1tERz+4cPOyu4N7JCc3tLWJsZ5q0Wao2AyIR07Dp/B3sv3EVWfiEAwEguw4C2jhj3XFN4u9mKHCUREemrxQf/xeGYFCiMiwZr23KwNkmU6IUFAEyePLncW5/Cw8NLbQsICEBAQEANR1V32ZibwMbcpswBYWqNgKSsfNy8l4O4+zmIv5eDuHu5iLv3AAlpuSgo1CA2ORuxydmljrWzUqC5vSXcHxYczeyKig9XWwu9W1k6R1mI03H3EXYlGWFXUnDvwX/jWxrbmGGYtwve6uoGRxuuRUFUFStWrMCCBQuQlJQELy8vLFu2DF26dCl3/23btuGzzz5DfHw8WrZsia+//hqDBg2qxYiJqteBK8lYdvgaAOCroe3RzpmDtUm6JFFYkP4wksvg/HCAd7cWdiVeK1RrcCcjDzdSc3A99UFRr0bqA9xIzUFKthL3HhQ9Hr1FqPicrg3M0dTOEk0bWsKtYdFtVk1sLeHSwPzhuBRxpeUUIOpWOs4nZODUjfs4n5CBQs1/M33VMzNGP08HDOvsgmebN6zTt4MRVZctW7Zg2rRpWL16Nbp27YolS5bAz88PsbGxaNSoUan9T5w4gTfeeAPBwcEYMmQINm3aBH9/f0RGRrJXm/TSnRxgxY6iScjHdW+GVzq5iBwR0ZOxsKBqY2wkh1tDS7g1tESf1iUb/ex8FeLu5eBG6sNi4+H/j7uXgzyVGvH3cxF/PxdAaqnzNqqngHODomLGqb45HK3NYGdpjBtZwM37uXBsYAlLU6Mqj11QqTVIyszHrfRc3E7Pw/XUB7ia/AD/Jmfjdnpeqf2b2FqgZys7+LV1RNdmDWFqrF+9LkRSt2jRIkyYMEE75m716tX4/fffsW7dOsycObPU/kuXLsWAAQMwY8YMAMAXX3yBsLAwLF++HKtXr67V2ImqQlmoxorD17HikhHUghrPNrfFrEEcrE3Sx8KCakU9MxN0cKmPDi71S2wXBAHJWUrcuPcAN+/nIv7h7VUJaXlIuJ+DnAK1dird8wkZj53VGEsvHwdQNLbD5uFaHvXMjGFhagwLUyMoTIxgLJdpZ7HSaASoBQH5KjVyH07pm5Gnwv0HBcjMe/LKw+72lujo2gA+TRugu7sdmjTU37UniKSuoKAAERERCAoK0m6Ty+Xo27cvTp48WeYxJ0+eLLFuEQD4+flh9+7d5V5HqVRCqfzvVsbi9TxUKpVOq5Efv3Yfey/exZ07chzdeanE2EB9otFomIMERNxMx417uQBkeM7dFgsDOkDQqKHSqMUOTSfF/4Z0+bckRYaQR1Vy0OUYFhYkKplMBkcbMzjamKGbe8nXBEFAWk4B7jycrepORh7uZuQjOSsfiZl5uJmcjlyNEfJURQsRpmYrkVrBtTzKY2osh0t9czg3MEfThpZo5WCFlg710MbRGjYWhjn4nEiK7t27B7VarZ3Io5iDgwNiYmLKPCYpKanM/ZOSksq9TnBwMObOnVtq+4EDB2BhUfEfD8ITZdgVbwRADqQkVvg4aWIOUlDPRMCrTTXo1DAFp/46KHY4VRIWFiZ2CNXCEPKoTA65ubkV3peFBUmWTCZDQysFGlopSvV0qFSqh4sV+qFAI0N6blGPQ2auCtnKQuQVqJFTUIiCQo12ZXQAMJIDcpkMZiZGsFQYwcLUGNZmJrCvZ4qGlgrYmJtwfARRHRIUFFSilyMrKwuurq7o378/rK2tK3wel9uZcLuaimvXrqJFi5Yw0tNfytUaDXOQAEuFMQZ62uHM8XD069dPbxewVKlUCAsL0+scAMPIoyo5FPfkVgQLC9J7uqzlQUT6wc7ODkZGRkhOTi6xPTk5GY6OjmUe4+joqNP+AKBQKKBQKEpt13X1cu9mdujgYoN9ef9iUJ8Wev3HB3OQhuLbT3T9b1GKDCEHwDDyqEwOuuyvn6U8EREZNFNTU3h7e+PQoUPabRqNBocOHYKvr2+Zx/j6+pbYHyjq9i9vfyIiql7ssSAiIkmaNm0aRo8eDR8fH3Tp0gVLlixBTk6OdpaoUaNGwdnZGcHBwQCADz/8EL169cLChQsxePBgbN68GefOncPatWvFTIOIqM5gYUFERJI0fPhwpKamYvbs2UhKSkLHjh2xf/9+7QDthISEErP+dOvWDZs2bcKnn36KWbNmoWXLlti9ezfXsCAiqiUsLIiISLImT56MyZMnl/laeHh4qW0BAQEICAio4aiIiKgsHGNBRERERERVxsKCiIiIiIiqrM7dCiUIResZ6DInbzGVSoXc3FxkZWXp9XRjhpAHc5AOQ8jDEHIAqpZH8Xdi8XdkXVXX2wjmIB2GkIch5AAYRh611T7UucIiOzsbAODq6ipyJERE0pOdnQ0bGxuxwxAN2wgiorJVpH2QCXXs5ymNRoO7d++iXr16kMl0W2G5eEXWW7du6bQiq9QYQh7MQToMIQ9DyAGoWh6CICA7OxtOTk4lZlqqa+p6G8EcpMMQ8jCEHADDyKO22oc612Mhl8vh4uJSpXNYW1vr7X9YjzKEPJiDdBhCHoaQA1D5POpyT0UxthFFmIN0GEIehpADYBh51HT7UHd/liIiIiIiomrDwoKIiIiIiKqMhYUOFAoF5syZA4VCIXYoVWIIeTAH6TCEPAwhB8Bw8tBXhvD+MwfpMIQ8DCEHwDDyqK0c6tzgbSIiIiIiqn7ssSAiIiIioipjYUFERERERFXGwoKIiIiIiKqMhUUlvfTSS2jSpAnMzMzQuHFjjBw5Enfv3hU7LJ3Ex8fj7bffRrNmzWBubg53d3fMmTMHBQUFYoemky+//BLdunWDhYUF6tevL3Y4FbZixQo0bdoUZmZm6Nq1K86cOSN2SDo5evQoXnzxRTg5OUEmk2H37t1ih6Sz4OBgPPPMM6hXrx4aNWoEf39/xMbGih2WTlatWoUOHTpo5yb39fXFH3/8IXZYdZ6+txGG0j4A+tlGsH0QnyG0D0DttxEsLCqpT58+2Lp1K2JjY7Fjxw5cv34dw4YNEzssncTExECj0WDNmjW4fPkyFi9ejNWrV2PWrFlih6aTgoICBAQEYNKkSWKHUmFbtmzBtGnTMGfOHERGRsLLywt+fn5ISUkRO7QKy8nJgZeXF1asWCF2KJX2119/ITAwEKdOnUJYWBhUKhX69++PnJwcsUOrMBcXF3z11VeIiIjAuXPn8Pzzz+Pll1/G5cuXxQ6tTtP3NsJQ2gdA/9oItg/SYAjtAyBCGyFQtdizZ48gk8mEgoICsUOpkm+++UZo1qyZ2GFUyvr16wUbGxuxw6iQLl26CIGBgdrnarVacHJyEoKDg0WMqvIACLt27RI7jCpLSUkRAAh//fWX2KFUSYMGDYQffvhB7DDoEYbQRuhz+yAI+tNGsH2QJkNpHwShZtsI9lhUg7S0NGzcuBHdunWDiYmJ2OFUSWZmJmxtbcUOw6AVFBQgIiICffv21W6Ty+Xo27cvTp48KWJklJmZCQB6+29ArVZj8+bNyMnJga+vr9jh0EOG0kawfah5bB+kS9/bB6B22ggWFlXwySefwNLSEg0bNkRCQgL27NkjdkhVcu3aNSxbtgzvvvuu2KEYtHv37kGtVsPBwaHEdgcHByQlJYkUFWk0GkyZMgXdu3dHu3btxA5HJ5cuXYKVlRUUCgUmTpyIXbt2wdPTU+yw6jxDaiPYPtQOtg/SpM/tA1C7bQQLi0fMnDkTMpnsiY+YmBjt/jNmzMD58+dx4MABGBkZYdSoURAksN6grnkAwJ07dzBgwAAEBARgwoQJIkX+n8rkQFQVgYGBiI6OxubNm8UORWceHh6IiorC6dOnMWnSJIwePRpXrlwROyyDYwhthCG0DwDbCKpd+tw+ALXbRnDl7Uekpqbi/v37T9ynefPmMDU1LbX99u3bcHV1xYkTJ0S/BUHXPO7evYvevXvj2WefRUhICORy8evNynwWISEhmDJlCjIyMmo4uqopKCiAhYUFtm/fDn9/f+320aNHIyMjQy9/1ZTJZNi1a1eJfPTJ5MmTsWfPHhw9ehTNmjUTO5wq69u3L9zd3bFmzRqxQzEohtBGGEL7ABhuG8H2QXoMrX0AaraNMK72M+oxe3t72NvbV+pYjUYDAFAqldUZUqXoksedO3fQp08feHt7Y/369ZJpNKryWUidqakpvL29cejQIe0XrUajwaFDhzB58mRxg6tjBEHA+++/j127diE8PNxgGg2NRiOJ7yJDYwhthCG0D4DhthFsH6TDUNsHoGbbCBYWlXD69GmcPXsWzz33HBo0aIDr16/js88+g7u7u+i9Fbq4c+cOevfuDTc3N3z77bdITU3Vvubo6ChiZLpJSEhAWloaEhISoFarERUVBQBo0aIFrKysxA2uHNOmTcPo0aPh4+ODLl26YMmSJcjJycHYsWPFDq3CHjx4gGvXrmmfx8XFISoqCra2tmjSpImIkVVcYGAgNm3ahD179qBevXrae5htbGxgbm4ucnQVExQUhIEDB6JJkybIzs7Gpk2bEB4ejj///FPs0OosQ2gjDKV9APSvjWD7IA2G0D4AIrQRNTLXlIG7ePGi0KdPH8HW1lZQKBRC06ZNhYkTJwq3b98WOzSdrF+/XgBQ5kOfjB49uswcjhw5InZoT7Rs2TKhSZMmgqmpqdClSxfh1KlTYoekkyNHjpT5vo8ePVrs0CqsvP/+169fL3ZoFTZu3DjBzc1NMDU1Fezt7YUXXnhBOHDggNhh1WmG0EYYSvsgCPrZRrB9EJ8htA+CUPttBMdYEBERERFRlUnnhkkiIiIiItJbLCyIiIiIiKjKWFgQEREREVGVsbAgIiIiIqIqY2FBRERERERVxsKCiIiIiIiqjIUFERERERFVGQsLIiIiIiKqMhYWRERERERUZSwsiIiIiIioylhYEBERERFRlbGwIKplqampcHR0xPz587XbTpw4AVNTUxw6dEjEyIiISExsH0jfyQRBEMQOgqiu2bdvH/z9/XHixAl4eHigY8eOePnll7Fo0SKxQyMiIhGxfSB9xsKCSCSBgYE4ePAgfHx8cOnSJZw9exYKhULssIiISGRsH0hfsbAgEkleXh7atWuHW7duISIiAu3btxc7JCIikgC2D6SvOMaCSCTXr1/H3bt3odFoEB8fL3Y4REQkEWwfSF+xx4JIBAUFBejSpQs6duwIDw8PLFmyBJcuXUKjRo3EDo2IiETE9oH0GQsLIhHMmDED27dvx4ULF2BlZYVevXrBxsYGe/fuFTs0IiISEdsH0me8FYqoloWHh2PJkiUIDQ2FtbU15HI5QkNDcezYMaxatUrs8IiISCRsH0jfsceCiIiIiIiqjD0WRERERERUZSwsiIiIiIioylhYEBERERFRlbGwICIiIiKiKmNhQUREREREVcbCgoiIiIiIqoyFBRERERERVRkLCyIiIiIiqjIWFkREREREVGUsLIiIiIiIqMpYWBARERERUZWxsCAiIiIioir7f//TFqXkmtRtAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 800x300 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "gelu, relu = GELU(), nn.ReLU()#先把函数给个小名\n",
    "\n",
    "# Some sample data\n",
    "x = torch.linspace(-3, 3, 100) #初定义一个张量\n",
    "y_gelu, y_relu = gelu(x), relu(x) #两种激活函数\n",
    "\n",
    "plt.figure(figsize=(8, 3))\n",
    "for i, (y, label) in enumerate(zip([y_gelu, y_relu], [\"GELU\", \"ReLU\"]), 1):\n",
    "    plt.subplot(1, 2, i)\n",
    "    plt.plot(x, y)\n",
    "    plt.title(f\"{label} activation function\")\n",
    "    plt.xlabel(\"x\")\n",
    "    plt.ylabel(f\"{label}(x)\")\n",
    "    plt.grid(True)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "#一个经典的作图"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cd01662-14cb-43fd-bffd-2d702813de2d",
   "metadata": {},
   "source": [
    "- 正如我们所见，ReLU 是一种分段线性函数：对于正值直接输出输入值；对于负值则输出零。\n",
    "- GELU 是一种平滑的非线性函数，它近似 ReLU，但在负值时具有非零梯度（除了大约在 -0.75 处）。\n",
    "\n",
    "- 接下来，我们将实现一个小型神经网络模块 `FeedForward`，该模块将用于 LLM 的 Transformer 块中："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "9275c879-b148-4579-a107-86827ca14d4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "class FeedForward(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.layers = nn.Sequential(\n",
    "            nn.Linear(cfg[\"emb_dim\"], 4 * cfg[\"emb_dim\"]),\n",
    "            GELU(),\n",
    "            nn.Linear(4 * cfg[\"emb_dim\"], cfg[\"emb_dim\"]),\n",
    "        )\n",
    "    #运行一次就线性两次激活一次\n",
    "    def forward(self, x):\n",
    "        return self.layers(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7c4976e2-0261-418e-b042-c5be98c2ccaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "768\n"
     ]
    }
   ],
   "source": [
    "print(GPT_CONFIG_124M[\"emb_dim\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdcaacfa-3cfc-4c9e-b668-b71a2753145a",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/09.webp?12\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "928e7f7c-d0b1-499f-8d07-4cadb428a6f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([2, 3, 768])\n"
     ]
    }
   ],
   "source": [
    "ffn = FeedForward(GPT_CONFIG_124M)\n",
    "\n",
    "# input shape: [batch_size, num_token, emb_size]\n",
    "x = torch.rand(2, 3, 768) \n",
    "out = ffn(x)\n",
    "print(out.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f8756c5-6b04-443b-93d0-e555a316c377",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/10.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5da2a50-04f4-4388-af23-ad32e405a972",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/11.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ffcb905-53c7-4886-87d2-4464c5fecf89",
   "metadata": {},
   "source": [
    "## 4.4 类似于ResNet的shortcut传递"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffae416c-821e-4bfa-a741-8af4ba5db00e",
   "metadata": {},
   "source": [
    "- 接下来，我们来讨论**快捷连接**（shortcut connections）的概念，也称为**跳跃连接**（skip connections）或**残差连接**（residual connections）。\n",
    "- 残差连接最初是在深度网络中提出的，主要应用于计算机视觉中的残差网络（ResNet），以缓解梯度消失问题。\n",
    "- 残差连接通过为梯度提供一条更短的替代路径，使其能够更顺畅地通过网络流动。\n",
    "- 具体实现是将某一层的输出与后续某一层的输出相加，通常会跳过中间的一层或多层。\n",
    "- 以下是一个小型网络的示例来说明这一概念：\n",
    "\n",
    "![残差连接示例](https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/12.webp?123)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14cfd241-a32e-4601-8790-784b82f2f23e",
   "metadata": {},
   "source": [
    "- 代码形式长成这样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "05473938-799c-49fd-86d4-8ed65f94fee6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class ExampleDeepNeuralNetwork(nn.Module):\n",
    "    def __init__(self, layer_sizes, use_shortcut):\n",
    "        super().__init__()\n",
    "        self.use_shortcut = use_shortcut\n",
    "        # 定义多层网络，包含 5 层线性层和激活函数 GELU\n",
    "        self.layers = nn.ModuleList([\n",
    "            nn.Sequential(nn.Linear(layer_sizes[0], layer_sizes[1]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[1], layer_sizes[2]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[2], layer_sizes[3]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[3], layer_sizes[4]), GELU()),\n",
    "            nn.Sequential(nn.Linear(layer_sizes[4], layer_sizes[5]), GELU())\n",
    "        ])\n",
    "        # 定义一个五层的神经网络块，其中每层包含一个线性变换和一个激活函数 GELU，\n",
    "        # 类似 ResNet 的结构，支持添加残差连接。\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 遍历每一层\n",
    "        for layer in self.layers:\n",
    "            # 当前层的输出\n",
    "            layer_output = layer(x)\n",
    "            # 检查是否可以应用残差连接\n",
    "            if self.use_shortcut and x.shape == layer_output.shape:\n",
    "                x = x + layer_output  # 如果输入和输出维度匹配，添加残差连接\n",
    "            else:\n",
    "                x = layer_output  # 否则直接输出当前层结果\n",
    "        return x  # 返回最终结果\n",
    "\n",
    "\n",
    "def print_gradients(model, x):\n",
    "    # 前向传播\n",
    "    output = model(x)\n",
    "    target = torch.tensor([[0.]])  # 定义目标值\n",
    "    # 计算损失，使用均方误差损失函数\n",
    "    loss = nn.MSELoss()\n",
    "    loss = loss(output, target)\n",
    "    \n",
    "    # 反向传播，计算梯度\n",
    "    loss.backward()\n",
    "\n",
    "    # 打印每层权重的梯度均值\n",
    "    for name, param in model.named_parameters():\n",
    "        if 'weight' in name:\n",
    "            print(f\"{name} has gradient mean of {param.grad.abs().mean().item()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b39bf277-b3db-4bb1-84ce-7a20caff1011",
   "metadata": {},
   "source": [
    "- 在没有残差链接的时候,梯度是这样的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c75f43cc-6923-4018-b980-26023086572c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layers.0.0.weight has gradient mean of 0.00020173587836325169\n",
      "layers.1.0.weight has gradient mean of 0.00012011159560643137\n",
      "layers.2.0.weight has gradient mean of 0.0007152039906941354\n",
      "layers.3.0.weight has gradient mean of 0.0013988736318424344\n",
      "layers.4.0.weight has gradient mean of 0.005049645435065031\n"
     ]
    }
   ],
   "source": [
    "layer_sizes = [3, 3, 3, 3, 3, 1]  \n",
    "\n",
    "sample_input = torch.tensor([[1., 0., -1.]])\n",
    "\n",
    "torch.manual_seed(123)\n",
    "model_without_shortcut = ExampleDeepNeuralNetwork(\n",
    "    layer_sizes, use_shortcut=False\n",
    ")\n",
    "print_gradients(model_without_shortcut, sample_input)\n",
    "#一次一次输出梯度"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "837fd5d4-7345-4663-97f5-38f19dfde621",
   "metadata": {},
   "source": [
    "- **有** 残差的链接,梯度是这样的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "11b7c0c2-f9dd-4dd5-b096-a05c48c5f6d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "layers.0.0.weight has gradient mean of 0.22169792652130127\n",
      "layers.1.0.weight has gradient mean of 0.20694106817245483\n",
      "layers.2.0.weight has gradient mean of 0.32896995544433594\n",
      "layers.3.0.weight has gradient mean of 0.2665732204914093\n",
      "layers.4.0.weight has gradient mean of 1.3258540630340576\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model_with_shortcut = ExampleDeepNeuralNetwork(\n",
    "    layer_sizes, use_shortcut=True\n",
    ")\n",
    "print_gradients(model_with_shortcut, sample_input)\n",
    "#引入了残差链接,发现梯度消失的缺点明显改善了"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79ff783a-46f0-49c5-a7a9-26a525764b6e",
   "metadata": {},
   "source": [
    "- 正如我们从上述输出中看到的，残差连接有效地防止了梯度在早期层（靠近 `layer.0`）中消失。\n",
    "- 接下来，在实现 Transformer 块时，我们将使用这一残差连接的概念。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae578ca-e564-42cf-8635-a2267047cdff",
   "metadata": {},
   "source": [
    "## 4.5 在 Transformer 块中连接attention层与线性层"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3daac6f-6545-4258-8f2d-f45a7394f429",
   "metadata": {},
   "source": [
    "- 在本节中，我们将把前面介绍的概念整合成一个所谓的 Transformer 块。\n",
    "- Transformer 块将上一章中的因果多头注意力模块与线性层以及我们之前实现的前馈神经网络相结合。\n",
    "- 此外，Transformer 块还包含 Dropout 和残差连接的机制。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0e1e8176-e5e3-4152-b1aa-0bbd7891dfd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from previous_chapters import MultiHeadAttention\n",
    "\n",
    "\n",
    "class TransformerBlock(nn.Module):\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.att = MultiHeadAttention(\n",
    "            d_in=cfg[\"emb_dim\"],               # 输入特征维度\n",
    "            d_out=cfg[\"emb_dim\"],              # 输出特征维度\n",
    "            context_length=cfg[\"context_length\"],  # 上下文长度\n",
    "            num_heads=cfg[\"n_heads\"],          # 注意力头的数量\n",
    "            dropout=cfg[\"drop_rate\"],          # Dropout 比例\n",
    "            qkv_bias=cfg[\"qkv_bias\"]           # 查询、键和值的偏置\n",
    "        )  # 多头注意力模块，结合各种参数\n",
    "        self.ff = FeedForward(cfg)  # 前馈神经网络模块\n",
    "        self.norm1 = LayerNorm(cfg[\"emb_dim\"])  # 第一归一化层\n",
    "        self.norm2 = LayerNorm(cfg[\"emb_dim\"])  # 第二归一化层\n",
    "        self.drop_shortcut = nn.Dropout(cfg[\"drop_rate\"])  # 残差连接的 Dropout\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 对注意力模块的快捷连接\n",
    "        shortcut = x\n",
    "        x = self.norm1(x)  # 应用第一归一化层\n",
    "        x = self.att(x)  # 通过多头注意力模块，形状为 [batch_size, num_tokens, emb_size]\n",
    "        x = self.drop_shortcut(x)  # 应用 Dropout\n",
    "        x = x + shortcut  # 将原始输入加回，实现残差连接\n",
    "\n",
    "        # 对前馈网络模块的残差连接\n",
    "        shortcut = x\n",
    "        x = self.norm2(x)  # 应用第二归一化层\n",
    "        x = self.ff(x)  # 通过前馈神经网络模块\n",
    "        x = self.drop_shortcut(x)  # 应用 Dropout\n",
    "        x = x + shortcut  # 将原始输入加回，实现残差连接\n",
    "\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36b64d16-94a6-4d13-8c85-9494c50478a9",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/13.webp?1\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54d2d375-87bd-4153-9040-63a1e6a2b7cb",
   "metadata": {},
   "source": [
    "- 假设我们有 2 个输入样本，每个样本包含 6 个标记，每个标记是一个 768 维的嵌入向量；然后，这个 Transformer 块会先应用自注意力机制，再通过线性层处理，生成一个相同尺寸的输出。\n",
    "- 你可以将输出视为我们在上一章讨论的上下文向量的增强版本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "3fb45a63-b1f3-4b08-b525-dafbc8228405",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input shape: torch.Size([2, 4, 768])\n",
      "Output shape: torch.Size([2, 4, 768])\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "\n",
    "x = torch.rand(2, 4, 768)  # Shape: [batch_size, num_tokens, emb_dim]\n",
    "block = TransformerBlock(GPT_CONFIG_124M)\n",
    "output = block(x)\n",
    "\n",
    "print(\"Input shape:\", x.shape)\n",
    "print(\"Output shape:\", output.shape)\n",
    "#经典的一系列操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f9e4ee4-cf23-4583-b1fd-317abb4fcd13",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/14.webp?1\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46618527-15ac-4c32-ad85-6cfea83e006e",
   "metadata": {},
   "source": [
    "## 4.6 编码GPT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dec7d03d-9ff3-4ca3-ad67-01b67c2f5457",
   "metadata": {},
   "source": [
    "- 终于要结束了：现在让我们将 Transformer 块插入到本章开头编写的架构中，这样就可以得到一个可用的 GPT 架构。\n",
    "- 请注意，Transformer 块会重复多次；对于最小的 124M GPT-2 模型来说，我们会重复 12 次。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b7b362d-f8c5-48d2-8ebd-722480ac5073",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/15.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "324e4b5d-ed89-4fdf-9a52-67deee0593bc",
   "metadata": {},
   "source": [
    "- 对应的代码实现，其中 `cfg[\"n_layers\"] = 12`："
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64e4e972-a13b-4733-9cf4-f54c882870cf",
   "metadata": {},
   "source": [
    "- (译者)一点语法小知识\n",
    "在 Python 中，* 是 解包操作符，用于将一个可迭代对象（如列表、元组）中的元素逐一解包成单独的参数。\n",
    "在 PyTorch 中，nn.Sequential 接受一组模块作为输入，而不是一个列表或其他容器。\n",
    "这里使用 * 将生成的 TransformerBlock 列表解包为独立的参数传递给 nn.Sequential。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c61de39c-d03c-4a32-8b57-f49ac3834857",
   "metadata": {},
   "outputs": [],
   "source": [
    "class GPTModel(nn.Module):#召唤GPT!\n",
    "    def __init__(self, cfg):\n",
    "        super().__init__()\n",
    "        self.tok_emb = nn.Embedding(cfg[\"vocab_size\"], cfg[\"emb_dim\"])\n",
    "        self.pos_emb = nn.Embedding(cfg[\"context_length\"], cfg[\"emb_dim\"])\n",
    "        self.drop_emb = nn.Dropout(cfg[\"drop_rate\"])\n",
    "        #新建字典、位置信息、还有dropout的比率设置\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "            *[TransformerBlock(cfg) for _ in range(cfg[\"n_layers\"])])\n",
    "        #解包操作\n",
    "\n",
    "        self.trf_blocks = nn.Sequential(\n",
    "        TransformerBlock(cfg),\n",
    "        TransformerBlock(cfg),\n",
    "        TransformerBlock(cfg)\n",
    "                )\n",
    "        self.final_norm = LayerNorm(cfg[\"emb_dim\"])\n",
    "        #归一化\n",
    "        self.out_head = nn.Linear(\n",
    "            cfg[\"emb_dim\"], cfg[\"vocab_size\"], bias=False\n",
    "        )\n",
    "        #输出头保证维度\n",
    "    def forward(self, in_idx):\n",
    "        batch_size, seq_len = in_idx.shape\n",
    "        tok_embeds = self.tok_emb(in_idx)\n",
    "        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))\n",
    "        x = tok_embeds + pos_embeds  # Shape [batch_size, num_tokens, emb_size]\n",
    "        x = self.drop_emb(x)\n",
    "        x = self.trf_blocks(x)\n",
    "        x = self.final_norm(x)\n",
    "        logits = self.out_head(x)\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2750270f-c45d-4410-8767-a6adbd05d5c3",
   "metadata": {},
   "source": [
    "- 用了124M模型的原始参数,我们接下啦要初始化模型参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "ef94fd9c-4e9d-470d-8f8e-dd23d1bb1f64",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input batch:\n",
      " tensor([[6109, 3626, 6100,  345],\n",
      "        [6109, 1110, 6622,  257]])\n",
      "\n",
      "Output shape: torch.Size([2, 4, 50257])\n",
      "tensor([[[ 0.3613,  0.4222, -0.0711,  ...,  0.3483,  0.4661, -0.2838],\n",
      "         [-0.1792, -0.5660, -0.9485,  ...,  0.0477,  0.5181, -0.3168],\n",
      "         [ 0.7120,  0.0332,  0.1085,  ...,  0.1018, -0.4327, -0.2553],\n",
      "         [-1.0076,  0.3418, -0.1190,  ...,  0.7195,  0.4023,  0.0532]],\n",
      "\n",
      "        [[-0.2564,  0.0900,  0.0335,  ...,  0.2659,  0.4454, -0.6806],\n",
      "         [ 0.1230,  0.3653, -0.2074,  ...,  0.7705,  0.2710,  0.2246],\n",
      "         [ 1.0558,  1.0318, -0.2800,  ...,  0.6936,  0.3205, -0.3178],\n",
      "         [-0.1565,  0.3926,  0.3288,  ...,  1.2630, -0.1858,  0.0388]]],\n",
      "       grad_fn=<UnsafeViewBackward0>)\n"
     ]
    }
   ],
   "source": [
    "torch.manual_seed(123)\n",
    "model = GPTModel(GPT_CONFIG_124M)\n",
    "\n",
    "out = model(batch)\n",
    "print(\"Input batch:\\n\", batch)\n",
    "print(\"\\nOutput shape:\", out.shape)\n",
    "print(out)\n",
    "#经典操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d616e7a-568b-4921-af29-bd3f4683cd2e",
   "metadata": {},
   "source": [
    "- 我们将在下一章训练这个模型。\n",
    "- 但关于其规模需要补充一点：我们之前提到它是一个拥有 1.24 亿参数的模型；我们可以通过以下方式再次确认这个数字："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "84fb8be4-9d3b-402b-b3da-86b663aac33a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of parameters: 163,009,536\n"
     ]
    }
   ],
   "source": [
    "total_params = sum(p.numel() for p in model.parameters())\n",
    "#模型的总参数数量\n",
    "print(f\"Total number of parameters: {total_params:,}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b67d13dd-dd01-4ba6-a2ad-31ca8a9fd660",
   "metadata": {},
   "source": [
    "- 正如我们上面看到的，这个模型实际上有 1.63 亿参数，而不是 1.24 亿；这是为什么呢？\n",
    "- 在原始 GPT-2 论文中，研究人员采用了**权重共享**（weight tying）技术，即将标记嵌入层（`tok_emb`）作为输出层复用，具体表现为设置 `self.out_head.weight = self.tok_emb.weight`。\n",
    "- 标记嵌入层将 50,257 维的独热编码输入标记映射到 768 维的嵌入表示。\n",
    "- 输出层则将 768 维的嵌入表示映射回 50,257 维的表示，从而可以将其还原为单词（关于这一点，我们将在下一节详细讨论）。\n",
    "- 因此，嵌入层和输出层的权重参数数量相同，从它们权重矩阵的形状可以看出这一点。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e3b43233-e9b8-4f5a-b72b-a263ec686982",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token embedding layer shape: torch.Size([50257, 768])\n",
      "Output layer shape: torch.Size([50257, 768])\n"
     ]
    }
   ],
   "source": [
    "print(\"Token embedding layer shape:\", model.tok_emb.weight.shape)\n",
    "print(\"Output layer shape:\", model.out_head.weight.shape)\n",
    "#输出格式让我们更好地理解"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f02259f6-6f79-4c89-a866-4ebeae1c3289",
   "metadata": {},
   "source": [
    "- 在原始 GPT-2 论文中，研究人员将标记嵌入矩阵复用为输出矩阵。\n",
    "- 相应地，如果我们减去输出层的参数数量，就会得到一个拥有 1.24 亿参数的模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "95a22e02-50d3-48b3-a4e0-d9863343c164",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of trainable parameters considering weight tying: 124,412,160\n"
     ]
    }
   ],
   "source": [
    "total_params_gpt2 =  total_params - sum(p.numel() for p in model.out_head.parameters())\n",
    "print(f\"Number of trainable parameters considering weight tying: {total_params_gpt2:,}\")\n",
    "#Parameter- sharing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40b03f80-b94c-46e7-9d42-d0df399ff3db",
   "metadata": {},
   "source": [
    "- 在实际应用中，因为不使用权重共享更容易训练模型，所以这里没有进行权重共享。\n",
    "- 不过，在第 5 章加载预训练权重时，我们将重新讨论并应用权重共享的概念。\n",
    "- 最后，我们可以通过以下方式计算模型的内存需求，这可能是一个有用的参考点："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5131a752-fab8-4d70-a600-e29870b33528",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total size of the model: 621.83 MB\n"
     ]
    }
   ],
   "source": [
    "# Calculate the total size in bytes (assuming float32, 4 bytes per parameter)\n",
    "total_size_bytes = total_params * 4\n",
    "\n",
    "# Convert to megabytes\n",
    "total_size_mb = total_size_bytes / (1024 * 1024)\n",
    "\n",
    "print(f\"Total size of the model: {total_size_mb:.2f} MB\")\n",
    "#计算总的容量"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "309a3be4-c20a-4657-b4e0-77c97510b47c",
   "metadata": {},
   "source": [
    "- **练习**：你可以尝试以下其他配置，这些配置参考自 [GPT-2 论文](https://scholar.google.com/citations?view_op=view_citation&hl=en&user=dOad5HoAAAAJ&citation_for_view=dOad5HoAAAAJ:YsMSGLbcyi4C)。\n",
    "\n",
    "    - **GPT2-small**（我们已经实现的 124M 配置）：\n",
    "        - `\"emb_dim\" = 768`\n",
    "        - `\"n_layers\" = 12`\n",
    "        - `\"n_heads\" = 12`\n",
    "\n",
    "    - **GPT2-medium**：\n",
    "        - `\"emb_dim\" = 1024`\n",
    "        - `\"n_layers\" = 24`\n",
    "        - `\"n_heads\" = 16`\n",
    "\n",
    "    - **GPT2-large**：\n",
    "        - `\"emb_dim\" = 1280`\n",
    "        - `\"n_layers\" = 36`\n",
    "        - `\"n_heads\" = 20`\n",
    "\n",
    "    - **GPT2-XL**：\n",
    "        - `\"emb_dim\" = 1600`\n",
    "        - `\"n_layers\" = 48`\n",
    "        - `\"n_heads\" = 25`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da5d9bc0-95ab-45d4-9378-417628d86e35",
   "metadata": {},
   "source": [
    "## 4.7 文本生成"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48da5deb-6ee0-4b9b-8dd2-abed7ed65172",
   "metadata": {},
   "source": [
    "- GPT架构的LLM一次只能生成一个单词"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caade12a-fe97-480f-939c-87d24044edff",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/16.webp\" width=\"400px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7061524-a3bd-4803-ade6-2e3b7b79ac13",
   "metadata": {},
   "source": [
    "- 以下的 `generate_text_simple` 函数实现了贪心解码（greedy decoding），这是一种简单且快速的文本生成方法。\n",
    "- 在贪心解码中，模型在每一步选择具有最高概率的词（或标记）作为下一个输出（由于最高的 logit 值对应最高的概率，实际上我们不需要显式地计算 softmax 函数）。\n",
    "- 在下一章，我们将实现一个更为复杂的 `generate_text` 函数。\n",
    "- 下图展示了给定输入上下文时，GPT 模型是如何生成下一个词标记的。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ee0f32c-c18c-445e-b294-a879de2aa187",
   "metadata": {},
   "source": [
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/17.webp\" width=\"600px\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c9b428a9-8764-4b36-80cd-7d4e00595ba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_text_simple(model, idx, max_new_tokens, context_size):\n",
    "    # 预测单词的模块\n",
    "    # idx 是当前上下文中的（batch, n_tokens）索引数组\n",
    "    for _ in range(max_new_tokens):\n",
    "        # 每次生成一个单词后，重新将其加入序列中\n",
    "        # 如果当前上下文长度超过模型支持的最大上下文长度，则截取\n",
    "        # 例如，如果LLM只支持5个token，而上下文长度为10\n",
    "        # 那么只使用最后5个token作为上下文\n",
    "        idx_cond = idx[:, -context_size:]\n",
    "        # 如果idx的长度超过模型支持的上下文长度size，只保留最后size个token\n",
    "        # 避免溢出\n",
    "        # 获取预测结果\n",
    "        with torch.no_grad():  # 在推理阶段，不需要计算梯度，因为没有反向传播\n",
    "            # 这样可以减少存储开销\n",
    "            logits = model(idx_cond)\n",
    "            # 模型输出结果\n",
    "        # 只关注最后一个时间步的输出\n",
    "        # (batch, n_tokens, vocab_size) 变为 (batch, vocab_size)\n",
    "        logits = logits[:, -1, :]\n",
    "        # 关注最后一个时间步\n",
    "        # 使用softmax函数计算概率\n",
    "        probas = torch.softmax(logits, dim=-1)  # (batch, vocab_size)\n",
    "        # 归一化\n",
    "        # 获取具有最高概率值的词汇索引\n",
    "        idx_next = torch.argmax(probas, dim=-1, keepdim=True)  # (batch, 1)\n",
    "        # 获取概率最高的词汇索引\n",
    "        # 将采样的索引添加到序列中\n",
    "        idx = torch.cat((idx, idx_next), dim=1)  # (batch, n_tokens+1)\n",
    "\n",
    "    return idx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6515f2c1-3cc7-421c-8d58-cc2f563b7030",
   "metadata": {},
   "source": [
    "- 上面的 `generate_text_simple` 实现了一个迭代过程，其中它一次生成一个token。\n",
    "\n",
    "<img src=\"https://sebastianraschka.com/images/LLMs-from-scratch-images/ch04_compressed/18.webp\" width=\"600px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f682eac4-f9bd-438b-9dec-6b1cc7bc05ce",
   "metadata": {},
   "source": [
    "- 举个例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3d7e3e94-df0f-4c0f-a6a1-423f500ac1d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "encoded: [15496, 11, 314, 716]\n",
      "encoded_tensor.shape: torch.Size([1, 4])\n"
     ]
    }
   ],
   "source": [
    "start_context = \"Hello, I am\"\n",
    "#模拟\n",
    "encoded = tokenizer.encode(start_context)\n",
    "print(\"encoded:\", encoded)\n",
    "#进行语义理解\n",
    "encoded_tensor = torch.tensor(encoded).unsqueeze(0)\n",
    "print(\"encoded_tensor.shape:\", encoded_tensor.shape)\n",
    "#最终输出格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "a72a9b60-de66-44cf-b2f9-1e638934ada4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output: tensor([[15496,    11,   314,   716, 27018, 24086, 47843, 30961, 42348,  7267]])\n",
      "Output length: 10\n"
     ]
    }
   ],
   "source": [
    "model.eval() # disable dropout\n",
    "#在检验的时候不需要正则化了\n",
    "out = generate_text_simple(\n",
    "    model=model,\n",
    "    #左边的参数名字,右边是函数传入的实际模型\n",
    "    idx=encoded_tensor, #上下文的索引\n",
    "    max_new_tokens=6, #最多运行六次,然后取结果概率最高的\n",
    "    #初始文本➕6\n",
    "    context_size=GPT_CONFIG_124M[\"context_length\"]\n",
    ")\n",
    "\n",
    "print(\"Output:\", out)\n",
    "print(\"Output length:\", len(out[0]))\n",
    "#输出长度还有每个单词的id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d131c00-1787-44ba-bec3-7c145497b2c3",
   "metadata": {},
   "source": [
    "- 去除批次维度并将其转换回文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "053d99f6-5710-4446-8d52-117fb34ea9f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, I am Featureiman Byeswickattribute argue\n"
     ]
    }
   ],
   "source": [
    "decoded_text = tokenizer.decode(out.squeeze(0).tolist())\n",
    "print(decoded_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a894003-51f6-4ccc-996f-3b9c7d5a1d70",
   "metadata": {},
   "source": [
    "- 请注意，目前模型尚未经过训练，因此上述输出文本是随机的。\n",
    "- 我们将在下一章中训练模型。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a35278b6-9e5c-480f-83e5-011a1173648f",
   "metadata": {},
   "source": [
    "## 总结与收获\n",
    "\n",
    "- 请查看 [./gpt.py](./gpt.py) 脚本，这是一个独立的脚本，包含了我们在此 Jupyter Notebook 中实现的 GPT 模型。\n",
    "- 练习题的解答可以在 [./exercise-solutions.ipynb](./exercise-solutions.ipynb) 中找到。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
